Oct 30 19:31:59.534243 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 19:31:59.534255 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 19:31:59.534264 ip-10-0-133-26 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 19:31:59.534584 ip-10-0-133-26 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 19:32:09.654299 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 19:32:09.654316 ip-10-0-133-26 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 57ece50741f24f0f8b5a703e85ac1e82 --
Oct 30 19:33:55.824734 ip-10-0-133-26 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:33:56.252628 ip-10-0-133-26 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:33:56.252628 ip-10-0-133-26 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:33:56.252628 ip-10-0-133-26 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:33:56.252628 ip-10-0-133-26 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:33:56.256212 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256121    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.256212 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256207    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.256212 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256212    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.256212 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256216    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.256212 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256220    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256225    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256231    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256235    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256239    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256242    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256246    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256249    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256253    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256257    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256261    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256265    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256268    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256272    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256276    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256280    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256284    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256289    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256294    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256297    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.256532 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256301    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256305    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256310    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256313    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256318    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256322    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256327    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256331    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256335    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256339    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256344    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256348    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256353    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256358    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256363    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256369    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256373    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256378    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256382    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.257356 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256386    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256390    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256395    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256399    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256404    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256408    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256413    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256417    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256421    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256426    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256430    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256435    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256439    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256443    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256448    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256452    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256456    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256460    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256480    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.258175 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256485    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256489    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256494    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256498    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256503    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256507    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256516    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256522    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256527    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256532    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256538    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256542    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256546    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256551    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256555    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256562    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256568    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256574    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.258976 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256579    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256583    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256587    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256592    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256596    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256601    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256605    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256609    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256614    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256619    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256623    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256628    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256632    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256636    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256641    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256646    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256650    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256656    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256660    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256664    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.259763 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256668    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256673    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256677    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256684    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256688    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.256694    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.257965    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.257975    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.257980    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.257984    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.257989    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.257994    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.257998    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258003    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258007    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258012    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258017    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258021    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258025    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258029    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.260671 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258033    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258038    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258042    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258047    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258051    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258055    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258060    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258065    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258069    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258075    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258080    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258084    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258088    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258093    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258097    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258104    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258110    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258115    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258121    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258126    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.261543 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258130    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258134    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258139    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258143    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258148    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258152    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258156    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258160    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258165    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258169    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258173    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258177    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258181    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258185    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258189    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258193    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258198    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258202    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258208    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258212    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.262374 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258216    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258223    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258227    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258231    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258236    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258240    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258244    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258249    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258253    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258258    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258262    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258267    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258272    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258276    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258280    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258284    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258288    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258292    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258300    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.262878 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258307    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258314    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258318    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258323    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258327    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258331    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258336    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258340    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258344    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258349    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258353    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258358    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258363    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258367    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258376    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258381    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258386    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258390    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258394    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258399    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.263537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258403    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258407    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258412    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258416    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258420    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258425    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258431    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258435    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258441    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258445    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258450    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258454    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.258459    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259304    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259313    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259318    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259323    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259327    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259331    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.264070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259336    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259340    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259345    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259351    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259355    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259359    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259363    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259371    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259376    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259380    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259384    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259388    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259392    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259397    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259401    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259405    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259410    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259414    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259420    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259424    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.264545 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259428    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259433    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259437    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259444    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259449    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259455    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259459    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259478    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259483    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259487    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259492    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259496    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259501    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259505    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259509    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259513    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259518    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259522    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259527    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.265117 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259533    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259537    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259542    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259546    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259550    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259554    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259558    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259563    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259567    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259571    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259575    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259580    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259585    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259590    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259594    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259599    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259603    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259608    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259613    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.265579 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259618    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259624    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259629    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259634    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259638    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259642    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259647    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259651    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259657    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259661    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259665    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259669    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259674    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259681    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259685    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259690    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259694    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259698    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259703    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.266038 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259707    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259711    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259715    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259719    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259724    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259728    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259732    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259745    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259749    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259754    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259758    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259762    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259766    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259770    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259774    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259778    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259781    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259786    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259790    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259794    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.266537 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259798    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259806    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.259810    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.259947    2586 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.259958    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.259967    2586 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.259975    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.259983    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.259989    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.259996    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260003    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260008    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260013    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260019    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260024    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260029    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260034    2586 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260039    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260044    2586 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260048    2586 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260064    2586 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260075    2586 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260080    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260086    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:33:56.266994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260091    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260096    2586 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260102    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260107    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260112    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260118    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260123    2586 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260130    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260135    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260140    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260145    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260151    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260157    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260162    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260168    2586 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260173    2586 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260178    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260188    2586 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260193    2586 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260199    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260204    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260209    2586 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260215    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260228    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260233    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:33:56.267571 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260238    2586 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260244    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260248    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260253    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260265    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260270    2586 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260276    2586 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260280    2586 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260287    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260292    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260297    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260302    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260307    2586 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260313    2586 flags.go:64] FLAG: --help="false"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260320    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260325    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260330    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260335    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260341    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260347    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260352    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260359    2586 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260365    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:33:56.268241 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260370    2586 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260376    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260382    2586 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260387    2586 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260392    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260396    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260401    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260407    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260412    2586 flags.go:64] FLAG: --lock-file=""
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260417    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260422    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260427    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260436    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260440    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260455    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260478    2586 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260485    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260490    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260495    2586 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260499    2586 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260506    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260511    2586 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260518    2586 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260525    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260530    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:33:56.268808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260534    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260539    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260544    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260549    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260553    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260567    2586 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260572    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260577    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260582    2586 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260587    2586 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260591    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260597    2586 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260602    2586 flags.go:64] FLAG: --port="10250"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260607    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260612    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b79a218db378e734"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260617    2586 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260622    2586 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260627    2586 flags.go:64] FLAG: --register-node="true"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260632    2586 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260638    2586 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260643    2586 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260647    2586 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260660    2586 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260666    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260671    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260676    2586 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:33:56.269428 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260680    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260685    2586 flags.go:64] FLAG: --runonce="false"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260689    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260694    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260701    2586 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260707    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260711    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260717    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260722    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260727    2586 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260732    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260739    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260744    2586 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260749    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260754    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260759    2586 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260764    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260772    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260777    2586 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260782    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260791    2586 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260796    2586 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260801    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260806    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260811    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:33:56.270542 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260816    2586 flags.go:64] FLAG: --v="2"
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260822    2586 flags.go:64] FLAG: --version="false"
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260829    2586 flags.go:64] FLAG: --vmodule=""
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260835    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.260842    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261075    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261083    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261087    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261092    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261096    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261101    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261108    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261113    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261117    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261122    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261126    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261130    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261135    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261141    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261146    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261150    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.271497 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261155    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261159    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261164    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261168    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261173    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261177    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261181    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261188    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261194    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261198    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261203    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261208    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261212    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261217    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261221    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261226    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261231    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261235    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261240    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261244    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.272184 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261249    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261253    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261260    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261265    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261269    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261273    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261277    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261282    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261286    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261293    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261298    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261303    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261307    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261311    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261316    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261320    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261324    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261329    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261333    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.273241 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261338    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261342    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261346    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261350    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261355    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261359    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261363    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261368    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261373    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261377    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261382    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261387    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261391    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261395    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261399    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261407    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261412    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261416    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261423    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.274042 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261429    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261434    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261439    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261445    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261450    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261454    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261458    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261478    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261483    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261487    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261492    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261496    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261500    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261504    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261508    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261513    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261517    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261521    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261526    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.274529 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261530    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261534    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261539    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261543    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261547    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261552    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261556    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261560    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261564    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261569    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261573    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261579    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261583    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.261625    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261686    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.275225 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261694    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261698    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261704    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261708    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261713    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261718    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261722    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261727    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261731    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261735    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261739    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261744    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261748    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261752    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261757    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261761    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261765    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261770    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261774    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261778    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.275636 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261782    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261786    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261791    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261796    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261801    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261805    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261810    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261818    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261824    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261829    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261834    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261838    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261844    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261849    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261853    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261858    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261862    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261866    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261870    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261874    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.276210 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261879    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261883    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261887    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261892    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261897    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261901    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261905    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261909    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261913    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261918    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261922    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261927    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261931    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261936    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261940    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261944    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261949    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261953    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261958    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.277091 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261962    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261966    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261970    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261975    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261979    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261983    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261987    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261993    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.261997    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262002    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262006    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262010    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262015    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262019    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262023    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262027    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262032    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262036    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262044    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.277572 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262050    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262054    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262059    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262063    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262067    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262071    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262076    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262080    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262084    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262088    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262093    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262098    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262102    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262106    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262111    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262115    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262119    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262123    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262127    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.278209 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262132    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262136    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262141    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262145    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262149    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262153    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262157    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.262161    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.271052    2586 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.271069    2586 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.271846    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.271862    2586 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271955    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271963    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271969    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271974    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271979    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271984    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271988    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.278673 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271993    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.271997    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272002    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272006    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272011    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272016    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272020    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272025    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272029    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272034    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272038    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272042    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272047    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272051    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272056    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272060    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272065    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272069    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272073    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272078    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.279181 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272082    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272096    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272102    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272106    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272110    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272115    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272119    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272123    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272127    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272132    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272137    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272141    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272145    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272150    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272156    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272160    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272164    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272169    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272173    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.279682 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272179    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272183    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272187    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272191    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272195    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272199    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272203    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272208    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272212    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272216    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272221    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272225    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272229    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272234    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272238    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272243    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272247    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272251    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272256    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272260    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.280208 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272265    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272270    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272274    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272278    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272283    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272288    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272292    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272297    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272303    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272310    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272315    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272319    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272324    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272329    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272334    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272338    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272343    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272347    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272352    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.280720 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272356    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272360    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272364    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272369    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272373    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272377    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272381    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272385    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272389    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272394    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272398    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272403    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272407    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272411    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272415    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272419    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272424    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272430    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272436    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.281378 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272441    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272446    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.272508    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272718    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272728    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272734    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272739    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272744    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272752    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272757    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272762    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272768    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272774    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272779    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272784    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272788    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.282154 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272793    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272797    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272802    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272806    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272810    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272814    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272819    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272824    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272829    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272833    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272838    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272842    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272847    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272851    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272855    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272860    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272864    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272868    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272872    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272877    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.282861 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272881    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272885    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272890    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272894    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272898    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272903    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272908    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272912    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272917    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272922    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272926    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272930    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272934    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272939    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272943    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272947    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272951    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272955    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272959    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.283589 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272964    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272968    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272972    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272976    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272981    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272985    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272989    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272993    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.272998    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273003    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273007    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273011    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273015    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273019    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273023    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273028    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273032    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273036    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273043    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273048    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.284024 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273054    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273058    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273062    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273066    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273071    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273075    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273079    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273084    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273088    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273092    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273097    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273101    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273105    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273110    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273116    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273121    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273126    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273131    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273135    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273140    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.284546 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273144    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273148    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273153    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273157    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273162    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273166    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273170    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273175    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273179    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273184    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273192    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273198    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273204    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273209    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.273247    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:33:56.285007 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273298    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273305    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273310    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273314    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273319    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273323    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273327    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273332    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273337    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273342    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273346    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273351    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273355    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273360    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273364    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273368    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273373    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273377    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273382    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:33:56.285416 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273386    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273391    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273395    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273399    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273404    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273408    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273412    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273417    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273422    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273426    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273431    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273436    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273441    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273445    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273450    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273454    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273458    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273477    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273482    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:33:56.285907 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273487    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273491    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273495    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273503    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273510    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273515    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273520    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273524    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273529    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273533    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273538    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273542    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273546    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273551    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273555    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273562    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273568    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273573    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273577    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:33:56.286379 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273582    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273587    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273592    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273597    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273601    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273606    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273611    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273615    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273620    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273624    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273628    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273632    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273637    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273641    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273645    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273650    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273654    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273658    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273663    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273667    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:33:56.286838 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273671    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273675    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273680    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273686    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273691    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273695    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273699    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273703    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273708    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273712    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273716    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273720    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273725    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273729    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273734    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273738    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273742    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273746    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273751    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:33:56.287479 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273756    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273760    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273764    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273768    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273772    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273776    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273780    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273785    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273789    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:56.273793    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.273870    2586 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.277739    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.278910    2586 server.go:1014] "Starting client certificate rotation"
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.279004    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.279036    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.281559    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:33:56.287925 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.283400    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:33:56.296300 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.296281    2586 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:33:56.300185 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.300172    2586 log.go:25] "Validated CRI v1 image API"
Oct 30 19:33:56.300976 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.300950    2586 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:33:56.303673 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.303654    2586 fs.go:135] Filesystem UUIDs: map[0c3270c1-f715-4db1-96b9-cc847717548f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ef613b7a-5eea-4767-aa65-e978e9b86cd9:/dev/nvme0n1p3]
Oct 30 19:33:56.303733 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.303671    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:33:56.310789 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.310433    2586 manager.go:217] Machine: {Timestamp:2026-10-30 19:33:56.308248248 +0000 UTC m=+0.385151149 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100655 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec241a90ba3e10b05d0b0554a511c352 SystemUUID:ec241a90-ba3e-10b0-5d0b-0554a511c352 BootID:57ece507-41f2-4f0f-8b5a-703e85ac1e82 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:9c:59:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:9c:59:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:4c:2c:08:14:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:33:56.310877 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.310789    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:33:56.310920 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.310903    2586 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:33:56.313565 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.313541    2586 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:33:56.313695 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.313566    2586 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:33:56.314418 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.314409    2586 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:33:56.314456 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.314419    2586 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:33:56.314456 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.314445    2586 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:33:56.314574 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.314459    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:33:56.315743 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.315733    2586 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:33:56.320730 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.320566    2586 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:33:56.320730 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.320715    2586 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:33:56.321579 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.321568    2586 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:33:56.321611 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.321585    2586 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:33:56.321611 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.321597    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:33:56.321611 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.321606    2586 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:33:56.321699 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.321615    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:33:56.321963 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.321944    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:33:56.324561 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.324543    2586 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:33:56.332002 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.331989    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:33:56.332894 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.332882    2586 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:33:56.332929 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.332913    2586 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:33:56.333077 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333066    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:33:56.333105 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333087    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:33:56.333105 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333097    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:33:56.333164 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333105    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:33:56.333164 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333143    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:33:56.333164 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333150    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:33:56.333804 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333795    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:33:56.333838 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333804    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:33:56.333838 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333813    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:33:56.333838 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333818    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:33:56.333838 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333827    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:33:56.333838 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.333835    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:33:56.334570 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.334562    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:33:56.334600 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.334572    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:33:56.338390 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.338370    2586 server.go:1263] "Started kubelet"
Oct 30 19:33:56.338491 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.338432    2586 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:33:56.338674 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.338543    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:33:56.338821 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.338800    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:33:56.339376 ip-10-0-133-26 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:33:56.340343 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.340310    2586 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:33:56.340877 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.340862    2586 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:33:56.347675 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.347657    2586 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:33:56.348639 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.348617    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:33:56.349967 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.349953    2586 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:33:56.350078 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.350052    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:56.350721 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.350702    2586 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:33:56.350826 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.350736    2586 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:33:56.350826 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.350707    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:33:56.350935 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.350908    2586 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:33:56.350935 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.350921    2586 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:33:56.351175 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.351152    2586 factory.go:55] Registering systemd factory
Oct 30 19:33:56.351175 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.351176    2586 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:33:56.351888 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.351871    2586 factory.go:153] Registering CRI-O factory
Oct 30 19:33:56.351888 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.351887    2586 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:33:56.351992 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.351936    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:33:56.351992 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.351951    2586 factory.go:103] Registering Raw factory
Oct 30 19:33:56.351992 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.351963    2586 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:33:56.353145 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.353128    2586 manager.go:319] Starting recovery of all containers
Oct 30 19:33:56.364910 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.364815    2586 manager.go:324] Recovery completed
Oct 30 19:33:56.366998 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.366978    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:33:56.367110 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.366979    2586 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:33:56.368232 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.367024    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-26.ec2.internal.18e365a516192619  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-26.ec2.internal,},FirstTimestamp:2026-10-30 19:33:56.338345497 +0000 UTC m=+0.415248398,LastTimestamp:2026-10-30 19:33:56.338345497 +0000 UTC m=+0.415248398,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-26.ec2.internal,}"
Oct 30 19:33:56.370871 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.370858    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:33:56.372445 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.372427    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:33:56.372528 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.372456    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:33:56.372528 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.372485    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:33:56.373031 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.373011    2586 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:33:56.373031 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.373027    2586 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:33:56.373149 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.373049    2586 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:33:56.373149 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.373028    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:33:56.373994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.373979    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:33:56.374073 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.374050    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:33:56.374073 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.374064    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:33:56.375351 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.375338    2586 policy_none.go:50] "Start"
Oct 30 19:33:56.375410 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.375353    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:33:56.375410 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.375363    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:33:56.376399 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.376384    2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:33:56.376475 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.376451    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8l6ts"
Oct 30 19:33:56.377370 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.377358    2586 policy_none.go:44] "Start"
Oct 30 19:33:56.382253 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.382227    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:33:56.382326 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.382230    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:33:56.382326 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.382289    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:33:56.385897 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.385881    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8l6ts"
Oct 30 19:33:56.400022 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.399938    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-26.ec2.internal.18e365a518216e78  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-26.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-26.ec2.internal,},FirstTimestamp:2026-10-30 19:33:56.372442744 +0000 UTC m=+0.449345645,LastTimestamp:2026-10-30 19:33:56.372442744 +0000 UTC m=+0.449345645,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-26.ec2.internal,}"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.420044    2586 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.422587    2586 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.423208    2586 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.423233    2586 server.go:87] "Starting device plugin registration server"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.423516    2586 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.423541    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.423671    2586 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.423764    2586 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.423772    2586 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.424422    2586 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.424487    2586 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.427184    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.428342    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.428369    2586 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.428397    2586 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:33:56.433898 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.428434    2586 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:33:56.444701 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.444687    2586 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:33:56.523989 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.523925    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:33:56.524893 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.524875    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:33:56.524970 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.524903    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:33:56.524970 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.524912    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:33:56.524970 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.524935    2586 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.529169 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.529016    2586 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"]
Oct 30 19:33:56.529247 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.529213    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:33:56.529960 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.529943    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:33:56.530037 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.529977    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:33:56.530037 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.529991    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:33:56.532286 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.532272    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:33:56.532440 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.532426    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.532504 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.532457    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:33:56.533022 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.533008    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:33:56.533022 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.533021    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:33:56.533152 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.533034    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:33:56.533152 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.533043    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:33:56.533152 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.533048    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:33:56.533152 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.533053    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:33:56.535126 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.535107    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.535208 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.535140    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:33:56.535972 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.535956    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:33:56.536043 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.535984    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:33:56.536043 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.535994    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:33:56.546976 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.546961    2586 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.547045 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.546982    2586 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-26.ec2.internal\": node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:56.552408 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.552395    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.552455 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.552417    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.552455 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.552434    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8c3ec59a3a8e5471fc2a192c7750dd6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"d8c3ec59a3a8e5471fc2a192c7750dd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.560856 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.560839    2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-26.ec2.internal\" not found" node="ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.562861 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.562844    2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-26.ec2.internal\" not found" node="ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.593390 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.593362    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:56.653162 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.653133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.653162 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.653164    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.653305 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.653189    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8c3ec59a3a8e5471fc2a192c7750dd6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"d8c3ec59a3a8e5471fc2a192c7750dd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.653305 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.653237    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8c3ec59a3a8e5471fc2a192c7750dd6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"d8c3ec59a3a8e5471fc2a192c7750dd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.653305 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.653252    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.653305 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.653252    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.694355 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.694325    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:56.795448 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.795388    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:56.866537 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.866510    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.868640 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:56.868623    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:56.895874 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.895839    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:56.996396 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:56.996355    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:57.097314 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.097240    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:57.197897 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.197855    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:33:57.227941 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.227908    2586 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:33:57.250813 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.250761    2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:57.278270 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.278241    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:33:57.278270 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.278263    2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:33:57.278809 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.278362    2586 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:33:57.278809 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.278398    2586 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:33:57.280412 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.280393    2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:33:57.322303 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.322274    2586 apiserver.go:52] "Watching apiserver"
Oct 30 19:33:57.344386 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.344361    2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:33:57.349431 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.349381    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:33:57.382168 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.382138    2586 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:33:57.382523 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.382499    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mb5hn","openshift-multus/multus-xs9fh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d","openshift-ovn-kubernetes/ovnkube-node-r5ssc","kube-system/global-pull-secret-syncer-htpxk","openshift-cluster-node-tuning-operator/tuned-rpc8d","openshift-dns/node-resolver-qmwpk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal","openshift-multus/multus-additional-cni-plugins-9mz4v","openshift-multus/network-metrics-daemon-ktz22","openshift-network-diagnostics/network-check-target-292kr","openshift-network-operator/iptables-alerter-hsdk9","kube-system/konnectivity-agent-zh9d6","kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"]
Oct 30 19:33:57.387076 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.386747    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.387845 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.387814    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:33:57.389670 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.389641    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:33:57.390165 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.390148    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.391516 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.391489    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:28:56 +0000 UTC" deadline="2028-05-25 23:29:15.171329515 +0000 UTC"
Oct 30 19:33:57.391516 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.391514    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13755h55m17.779818395s"
Oct 30 19:33:57.392559 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.392524    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.395560 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.395526    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.398327 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.397872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.400267 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.400247    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.402790 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.402768    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.404381 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.404363    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxzn8\""
Oct 30 19:33:57.404490 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.404478    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:33:57.404663 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.404649    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:33:57.405861 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.405842    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.407249 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.407227    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fqfcc\""
Oct 30 19:33:57.407349 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.407278    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:33:57.407418 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.407388    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:33:57.408443 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.408424    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:57.408528 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.408474    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:33:57.408528 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.408483    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:33:57.411565 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.411393    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.415198 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.415176    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.415294 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.415253    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:33:57.418639 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.418616    2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:33:57.418766 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.418687    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mtpsd"
Oct 30 19:33:57.451704 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.451680    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:33:57.457984 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.457963    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-host\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.458086 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.457994    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-serviceca\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.458086 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458012    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-cni-binary-copy\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.458086 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458040    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-netns\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.458212 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458106    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-multus\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.458212 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458146    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.458212 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458176    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0403f76e-6189-4e37-bcb0-b8cecd653283-konnectivity-ca\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.458212 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458202    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-hostroot\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.458404 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458229    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:33:57.458404 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458291    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-systemd\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.458404 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458329    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cnibin\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.458404 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458356    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.458404 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458381    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-tmp-dir\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.458650 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458408    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffflj\" (UniqueName: \"kubernetes.io/projected/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-kube-api-access-ffflj\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.458650 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458462    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-conf-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.458650 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458515    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-multus-certs\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.458650 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458544    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-etc-selinux\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.458650 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458614    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.458837 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v762g\" (UniqueName: \"kubernetes.io/projected/4e4bc0c6-9f96-4687-a645-5b727436c653-kube-api-access-v762g\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:57.458837 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458686    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-systemd-units\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.458837 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458709    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.458837 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458755    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-netd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.458837 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8r5\" (UniqueName: \"kubernetes.io/projected/734659d4-c70c-40be-8232-3f2d844704c1-kube-api-access-lx8r5\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.458837 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458793    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-modprobe-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.458837 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458821    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-run\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458862    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458893    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-hosts-file\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458918    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-cnibin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458952    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-sys-fs\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.458979    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-netns\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459002    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-lib-modules\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459040    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-tmp\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459069    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-system-cni-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459097    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459153    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmrvv\" (UniqueName: \"kubernetes.io/projected/b290737d-48ee-4120-a1c8-46917d8f982c-kube-api-access-hmrvv\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.459193 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459182    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdch\" (UniqueName: \"kubernetes.io/projected/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-kube-api-access-ltdch\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459198    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-multus-daemon-config\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-device-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459279    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9v9m\" (UniqueName: \"kubernetes.io/projected/c37d572e-f147-442e-8b80-18a23ed12528-kube-api-access-r9v9m\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459300    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-slash\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459328    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-etc-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysconfig\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-kubernetes\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459384    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-var-lib-kubelet\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459415    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-tuned\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459433    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b290737d-48ee-4120-a1c8-46917d8f982c-iptables-alerter-script\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459456    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b290737d-48ee-4120-a1c8-46917d8f982c-host-slash\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459525    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-bin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459559    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-etc-kubernetes\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459589    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459616    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-systemd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.459681 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459643    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-var-lib-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459684    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-config\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459722    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-kubelet-config\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459783    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459812    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-env-overrides\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459844    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-sys\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459874    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phw2p\" (UniqueName: \"kubernetes.io/projected/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-kube-api-access-phw2p\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459904    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0403f76e-6189-4e37-bcb0-b8cecd653283-agent-certs\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459929    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-socket-dir-parent\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459957    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-kubelet\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459982    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htz2\" (UniqueName: \"kubernetes.io/projected/5f9691d9-c7cd-4b98-8003-39cc09d90a02-kube-api-access-2htz2\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.459999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-k8s-cni-cncf-io\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460043    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-log-socket\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460087    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460117    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-dbus\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460150    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-socket-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460175    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460204    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/734659d4-c70c-40be-8232-3f2d844704c1-ovn-node-metrics-cert\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460235    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-conf\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460268    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-host\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460304    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460339    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-system-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460367    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwfql\" (UniqueName: \"kubernetes.io/projected/9d96c176-1899-4867-8855-09bfb02134e2-kube-api-access-nwfql\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460391    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-registration-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-kubelet\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460439    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-ovn\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460488    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-node-log\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460540    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460578    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-bin\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.460957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.460621    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-script-lib\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.505823 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.505782    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c3ec59a3a8e5471fc2a192c7750dd6.slice/crio-faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9 WatchSource:0}: Error finding container faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9: Status 404 returned error can't find the container with id faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9
Oct 30 19:33:57.506182 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.506162    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e03f428e399a396d4b35504155fa47.slice/crio-47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63 WatchSource:0}: Error finding container 47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63: Status 404 returned error can't find the container with id 47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63
Oct 30 19:33:57.513040 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513004    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:33:57.513174 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513157    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:33:57.513447 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513427    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:33:57.513447 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513429    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4c78\""
Oct 30 19:33:57.513725 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513700    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t9cqt\""
Oct 30 19:33:57.513825 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513712    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:33:57.513825 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513800    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:33:57.513825 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513804    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:33:57.513996 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513850    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lhsbg\""
Oct 30 19:33:57.513996 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513875    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:33:57.513996 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513853    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:33:57.513996 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.513875    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:33:57.514190 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514149    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vj26l\""
Oct 30 19:33:57.514190 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514173    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:33:57.514190 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514177    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514176    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514201    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514228    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514228    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xhvhw\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514248    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514203    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514266    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514250    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514289    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qv89v\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514305    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514340    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:33:57.514346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514351    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:33:57.514943 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514353    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:33:57.514943 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.514822    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4mr2r\""
Oct 30 19:33:57.515505 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.515488    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:33:57.560863 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-k8s-cni-cncf-io\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.560863 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560862    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-log-socket\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560882    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560903    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-dbus\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560948    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-socket-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560950    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-k8s-cni-cncf-io\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560967    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-log-socket\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.560973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561024    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/734659d4-c70c-40be-8232-3f2d844704c1-ovn-node-metrics-cert\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561050    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-conf\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-host\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561083    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-socket-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561097    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.561122 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561095    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561142    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-dbus\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561148    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561144    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-system-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561144    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-host\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwfql\" (UniqueName: \"kubernetes.io/projected/9d96c176-1899-4867-8855-09bfb02134e2-kube-api-access-nwfql\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561211    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-system-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561225    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-registration-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561250    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-conf\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561262    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-kubelet\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561269    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-registration-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561308    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-kubelet\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561333    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-ovn\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561362    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-node-log\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561393    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561415    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-bin\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561392    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-ovn\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-script-lib\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.561649 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561449    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-node-log\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561464    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-host\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561498    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-bin\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561545    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-host\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561559    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-serviceca\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561689    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-cni-binary-copy\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561748    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-netns\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561849    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-netns\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561879    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-multus\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561904    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-multus\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561922    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561969    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.561971    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0403f76e-6189-4e37-bcb0-b8cecd653283-konnectivity-ca\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562005    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-hostroot\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562031    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-systemd\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cnibin\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.562291 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562100    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562110    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-systemd\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-tmp-dir\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562149    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cnibin\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562182    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffflj\" (UniqueName: \"kubernetes.io/projected/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-kube-api-access-ffflj\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562212    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-conf-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562240    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-multus-certs\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-etc-selinux\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562295    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562322    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v762g\" (UniqueName: \"kubernetes.io/projected/4e4bc0c6-9f96-4687-a645-5b727436c653-kube-api-access-v762g\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562341    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562351    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-systemd-units\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562386    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-multus-certs\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562387    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562425    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-netd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562448    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562459    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8r5\" (UniqueName: \"kubernetes.io/projected/734659d4-c70c-40be-8232-3f2d844704c1-kube-api-access-lx8r5\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563172 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-modprobe-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562059    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-hostroot\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562637    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-etc-selinux\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562652    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-tmp-dir\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562670    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-modprobe-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562748    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-systemd-units\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562761    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-netd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562828    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-conf-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562881    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-serviceca\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562884    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-run\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.562916    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563049    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-run\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563048    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-script-lib\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563068    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563141    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-hosts-file\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-cnibin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.563237    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563304    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-hosts-file\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.563923 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563307    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-sys-fs\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563289    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.563341    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret podName:c16bcfc9-20dd-4a12-ab7b-79dcbcb12557 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:58.063319182 +0000 UTC m=+2.140222085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret") pod "global-pull-secret-syncer-htpxk" (UID: "c16bcfc9-20dd-4a12-ab7b-79dcbcb12557") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563688    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-netns\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563705    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-sys-fs\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563521    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-cnibin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563734    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-lib-modules\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563735    2586 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563752    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-cni-binary-copy\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563767    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-tmp\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563846    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-netns\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563892    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-system-cni-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563928    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-lib-modules\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563932    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563965    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrvv\" (UniqueName: \"kubernetes.io/projected/b290737d-48ee-4120-a1c8-46917d8f982c-kube-api-access-hmrvv\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.563985    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-system-cni-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564003    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdch\" (UniqueName: \"kubernetes.io/projected/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-kube-api-access-ltdch\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.564781 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564038    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-multus-daemon-config\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-device-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9v9m\" (UniqueName: \"kubernetes.io/projected/c37d572e-f147-442e-8b80-18a23ed12528-kube-api-access-r9v9m\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564141    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-slash\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564174    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-etc-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564214    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysconfig\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-kubernetes\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564337    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-var-lib-kubelet\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564340    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0403f76e-6189-4e37-bcb0-b8cecd653283-konnectivity-ca\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564431    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-slash\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564453    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-etc-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564480    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysconfig\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564534    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564277    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-var-lib-kubelet\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564593    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-tuned\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564616    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b290737d-48ee-4120-a1c8-46917d8f982c-iptables-alerter-script\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564629    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-kubernetes\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.565629 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564642    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b290737d-48ee-4120-a1c8-46917d8f982c-host-slash\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564673    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-bin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564741    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b290737d-48ee-4120-a1c8-46917d8f982c-host-slash\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564847    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-etc-kubernetes\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564922    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-systemd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-var-lib-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.564983    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-config\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-kubelet-config\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-env-overrides\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565071    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-multus-daemon-config\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565173    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-sys\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phw2p\" (UniqueName: \"kubernetes.io/projected/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-kube-api-access-phw2p\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565221    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-var-lib-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0403f76e-6189-4e37-bcb0-b8cecd653283-agent-certs\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565273    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-socket-dir-parent\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.565359    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:33:57.566453 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565385    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-etc-kubernetes\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565408    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-kubelet\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htz2\" (UniqueName: \"kubernetes.io/projected/5f9691d9-c7cd-4b98-8003-39cc09d90a02-kube-api-access-2htz2\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565522    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-bin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565659    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-sys\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565689    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-systemd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565854    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-kubelet-config\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.565951    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.566117    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-kubelet\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.566213    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-socket-dir-parent\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.566285    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:58.066264525 +0000 UTC m=+2.143167423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:33:57.567376 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.566508    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-device-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.568207 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.568175    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b290737d-48ee-4120-a1c8-46917d8f982c-iptables-alerter-script\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.568878 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.568536    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-tmp\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.568878 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.568853    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/734659d4-c70c-40be-8232-3f2d844704c1-ovn-node-metrics-cert\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.569981 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.569958    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-env-overrides\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.570068 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.569988    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-tuned\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.570068 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.570042    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-config\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.570478 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.570459    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0403f76e-6189-4e37-bcb0-b8cecd653283-agent-certs\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.588560 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.588533    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:33:57.588560 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.588558    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:33:57.588686 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.588570    2586 projected.go:196] Error preparing data for projected volume kube-api-access-prfv4 for pod openshift-network-diagnostics/network-check-target-292kr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:33:57.588686 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:57.588625    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4 podName:28adba45-6c1c-4151-a898-94eaf4974f81 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:58.088611044 +0000 UTC m=+2.165513955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-prfv4" (UniqueName: "kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4") pod "network-check-target-292kr" (UID: "28adba45-6c1c-4151-a898-94eaf4974f81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:33:57.589043 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.589024    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwfql\" (UniqueName: \"kubernetes.io/projected/9d96c176-1899-4867-8855-09bfb02134e2-kube-api-access-nwfql\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.591032 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.591003    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v762g\" (UniqueName: \"kubernetes.io/projected/4e4bc0c6-9f96-4687-a645-5b727436c653-kube-api-access-v762g\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:57.591667 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.591614    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrvv\" (UniqueName: \"kubernetes.io/projected/b290737d-48ee-4120-a1c8-46917d8f982c-kube-api-access-hmrvv\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.593038 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.593014    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9v9m\" (UniqueName: \"kubernetes.io/projected/c37d572e-f147-442e-8b80-18a23ed12528-kube-api-access-r9v9m\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.593184 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.593163    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8r5\" (UniqueName: \"kubernetes.io/projected/734659d4-c70c-40be-8232-3f2d844704c1-kube-api-access-lx8r5\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.593240 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.593176    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htz2\" (UniqueName: \"kubernetes.io/projected/5f9691d9-c7cd-4b98-8003-39cc09d90a02-kube-api-access-2htz2\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.593240 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.593179    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdch\" (UniqueName: \"kubernetes.io/projected/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-kube-api-access-ltdch\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.593381 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.593365    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffflj\" (UniqueName: \"kubernetes.io/projected/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-kube-api-access-ffflj\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.594665 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.594648    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phw2p\" (UniqueName: \"kubernetes.io/projected/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-kube-api-access-phw2p\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.705045 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.705015    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:33:57.716070 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.716042    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a47a9c2_1815_4bd6_b1bd_eb047f1cf679.slice/crio-7ae66028480815792c693695be196914e95bdeacc4913ce4470cfa5013875106 WatchSource:0}: Error finding container 7ae66028480815792c693695be196914e95bdeacc4913ce4470cfa5013875106: Status 404 returned error can't find the container with id 7ae66028480815792c693695be196914e95bdeacc4913ce4470cfa5013875106
Oct 30 19:33:57.719460 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.719442    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:33:57.729335 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.729306    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37d572e_f147_442e_8b80_18a23ed12528.slice/crio-4f77ce3f3d68a73ad1c89fa3da4f05cf08e65255b69896c984c2eef9c68187d3 WatchSource:0}: Error finding container 4f77ce3f3d68a73ad1c89fa3da4f05cf08e65255b69896c984c2eef9c68187d3: Status 404 returned error can't find the container with id 4f77ce3f3d68a73ad1c89fa3da4f05cf08e65255b69896c984c2eef9c68187d3
Oct 30 19:33:57.741063 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.741044    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:33:57.741245 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.741225    2586 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:33:57.749162 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.749137    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734659d4_c70c_40be_8232_3f2d844704c1.slice/crio-c620f89f08c9530f2397517431a8ee3ec99ce67a904cada33f82c575f2cddf2f WatchSource:0}: Error finding container c620f89f08c9530f2397517431a8ee3ec99ce67a904cada33f82c575f2cddf2f: Status 404 returned error can't find the container with id c620f89f08c9530f2397517431a8ee3ec99ce67a904cada33f82c575f2cddf2f
Oct 30 19:33:57.749251 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.749178    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:33:57.755447 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.755430    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:33:57.757631 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.757607    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb290737d_48ee_4120_a1c8_46917d8f982c.slice/crio-3ea660b33bb399f60b597aa0552a5287568c5ea36bcee75969e142d6a5297dfa WatchSource:0}: Error finding container 3ea660b33bb399f60b597aa0552a5287568c5ea36bcee75969e142d6a5297dfa: Status 404 returned error can't find the container with id 3ea660b33bb399f60b597aa0552a5287568c5ea36bcee75969e142d6a5297dfa
Oct 30 19:33:57.761061 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.761041    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:33:57.771830 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.771809    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfd71eb_3976_4e6f_8bf2_182a1c9f22b5.slice/crio-072fe2ae46557f05ebf9b841780c4d07797e4fab742e614ad7a3a18ed7ebeec8 WatchSource:0}: Error finding container 072fe2ae46557f05ebf9b841780c4d07797e4fab742e614ad7a3a18ed7ebeec8: Status 404 returned error can't find the container with id 072fe2ae46557f05ebf9b841780c4d07797e4fab742e614ad7a3a18ed7ebeec8
Oct 30 19:33:57.791323 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.791304    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:33:57.800389 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.800364    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0403f76e_6189_4e37_bcb0_b8cecd653283.slice/crio-af9b796f616f8513c134b681bfeea6e16769b0ed5e37422fb8c0e8be8374b279 WatchSource:0}: Error finding container af9b796f616f8513c134b681bfeea6e16769b0ed5e37422fb8c0e8be8374b279: Status 404 returned error can't find the container with id af9b796f616f8513c134b681bfeea6e16769b0ed5e37422fb8c0e8be8374b279
Oct 30 19:33:57.816552 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.816527    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xs9fh"
Oct 30 19:33:57.825554 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.825534    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96c176_1899_4867_8855_09bfb02134e2.slice/crio-d2ab193b546d6e627434bc6df3032bb57f82b33beeec27ad45063d9f1c9be0bb WatchSource:0}: Error finding container d2ab193b546d6e627434bc6df3032bb57f82b33beeec27ad45063d9f1c9be0bb: Status 404 returned error can't find the container with id d2ab193b546d6e627434bc6df3032bb57f82b33beeec27ad45063d9f1c9be0bb
Oct 30 19:33:57.835832 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.835813    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:33:57.845141 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:33:57.845121    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c72a9e_1e40_4f4d_ae0f_b42a4923c634.slice/crio-e1f3687bc80427c23e2fdfedf273d0e3d7a88794452e3eeede25087e729ed231 WatchSource:0}: Error finding container e1f3687bc80427c23e2fdfedf273d0e3d7a88794452e3eeede25087e729ed231: Status 404 returned error can't find the container with id e1f3687bc80427c23e2fdfedf273d0e3d7a88794452e3eeede25087e729ed231
Oct 30 19:33:57.956994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:57.956924    2586 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:33:58.069888 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.069852    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:58.070052 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.069909    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:58.070052 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.070013    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:33:58.070052 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.070046    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:33:58.070185 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.070086    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret podName:c16bcfc9-20dd-4a12-ab7b-79dcbcb12557 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:59.070065987 +0000 UTC m=+3.146968888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret") pod "global-pull-secret-syncer-htpxk" (UID: "c16bcfc9-20dd-4a12-ab7b-79dcbcb12557") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:33:58.070185 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.070103    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:59.070095945 +0000 UTC m=+3.146998834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:33:58.171375 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.171337    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:33:58.171571 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.171528    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:33:58.171571 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.171550    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:33:58.171571 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.171566    2586 projected.go:196] Error preparing data for projected volume kube-api-access-prfv4 for pod openshift-network-diagnostics/network-check-target-292kr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:33:58.171801 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:58.171637    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4 podName:28adba45-6c1c-4151-a898-94eaf4974f81 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:59.17161762 +0000 UTC m=+3.248520530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-prfv4" (UniqueName: "kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4") pod "network-check-target-292kr" (UID: "28adba45-6c1c-4151-a898-94eaf4974f81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:33:58.422114 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.421975    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:28:57 +0000 UTC" deadline="2028-05-23 10:29:14.187605895 +0000 UTC"
Oct 30 19:33:58.422114 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.422047    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13694h55m15.765563007s"
Oct 30 19:33:58.444567 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.444454    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mb5hn" event={"ID":"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5","Type":"ContainerStarted","Data":"072fe2ae46557f05ebf9b841780c4d07797e4fab742e614ad7a3a18ed7ebeec8"}
Oct 30 19:33:58.463511 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.463450    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d" event={"ID":"c37d572e-f147-442e-8b80-18a23ed12528","Type":"ContainerStarted","Data":"4f77ce3f3d68a73ad1c89fa3da4f05cf08e65255b69896c984c2eef9c68187d3"}
Oct 30 19:33:58.473562 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.470950    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qmwpk" event={"ID":"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679","Type":"ContainerStarted","Data":"7ae66028480815792c693695be196914e95bdeacc4913ce4470cfa5013875106"}
Oct 30 19:33:58.474829 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.474803    2586 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:33:58.474950 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.474836    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" event={"ID":"d8c3ec59a3a8e5471fc2a192c7750dd6","Type":"ContainerStarted","Data":"faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9"}
Oct 30 19:33:58.479012 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.478970    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zh9d6" event={"ID":"0403f76e-6189-4e37-bcb0-b8cecd653283","Type":"ContainerStarted","Data":"af9b796f616f8513c134b681bfeea6e16769b0ed5e37422fb8c0e8be8374b279"}
Oct 30 19:33:58.485181 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.485085    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpc8d" event={"ID":"5f9691d9-c7cd-4b98-8003-39cc09d90a02","Type":"ContainerStarted","Data":"e9296daf9d613272919bcac26229bf77a7a32dddbf9dddd7d39cb36bb31a802a"}
Oct 30 19:33:58.493584 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.493488    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hsdk9" event={"ID":"b290737d-48ee-4120-a1c8-46917d8f982c","Type":"ContainerStarted","Data":"3ea660b33bb399f60b597aa0552a5287568c5ea36bcee75969e142d6a5297dfa"}
Oct 30 19:33:58.500799 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.500713    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"c620f89f08c9530f2397517431a8ee3ec99ce67a904cada33f82c575f2cddf2f"}
Oct 30 19:33:58.505253 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.505189    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"76e03f428e399a396d4b35504155fa47","Type":"ContainerStarted","Data":"47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63"}
Oct 30 19:33:58.511088 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.511033    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerStarted","Data":"e1f3687bc80427c23e2fdfedf273d0e3d7a88794452e3eeede25087e729ed231"}
Oct 30 19:33:58.527645 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:58.527600    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xs9fh" event={"ID":"9d96c176-1899-4867-8855-09bfb02134e2","Type":"ContainerStarted","Data":"d2ab193b546d6e627434bc6df3032bb57f82b33beeec27ad45063d9f1c9be0bb"}
Oct 30 19:33:59.079459 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.079249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:59.079459 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.079311    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:59.079459 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.079462    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:33:59.079789 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.079543    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:01.079521701 +0000 UTC m=+5.156424606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:33:59.080141 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.079984    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:33:59.080141 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.080041    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret podName:c16bcfc9-20dd-4a12-ab7b-79dcbcb12557 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:01.080025033 +0000 UTC m=+5.156927924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret") pod "global-pull-secret-syncer-htpxk" (UID: "c16bcfc9-20dd-4a12-ab7b-79dcbcb12557") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:33:59.180944 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.180170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:33:59.180944 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.180439    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:33:59.180944 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.180460    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:33:59.180944 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.180473    2586 projected.go:196] Error preparing data for projected volume kube-api-access-prfv4 for pod openshift-network-diagnostics/network-check-target-292kr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:33:59.180944 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.180561    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4 podName:28adba45-6c1c-4151-a898-94eaf4974f81 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:01.180540541 +0000 UTC m=+5.257443448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-prfv4" (UniqueName: "kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4") pod "network-check-target-292kr" (UID: "28adba45-6c1c-4151-a898-94eaf4974f81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:33:59.422679 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.422631    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:28:57 +0000 UTC" deadline="2028-08-11 09:06:22.9455323 +0000 UTC"
Oct 30 19:33:59.422679 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.422676    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15613h32m23.522860331s"
Oct 30 19:33:59.429158 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.429133    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:33:59.429295 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.429265    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:33:59.429743 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.429718    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:33:59.429859 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.429830    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:33:59.429928 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:33:59.429915    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:33:59.430025 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:33:59.430004    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:00.020310 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:00.020240    2586 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:34:01.100056 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:01.099131    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:01.100056 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:01.099199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:01.100056 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.099374    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:34:01.100056 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.099440    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:05.099421306 +0000 UTC m=+9.176324212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:34:01.100056 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.099923    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:34:01.100056 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.100003    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret podName:c16bcfc9-20dd-4a12-ab7b-79dcbcb12557 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:05.09996384 +0000 UTC m=+9.176866745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret") pod "global-pull-secret-syncer-htpxk" (UID: "c16bcfc9-20dd-4a12-ab7b-79dcbcb12557") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:34:01.200839 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:01.199975    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:01.200839 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.200203    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:34:01.200839 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.200227    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:34:01.200839 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.200242    2586 projected.go:196] Error preparing data for projected volume kube-api-access-prfv4 for pod openshift-network-diagnostics/network-check-target-292kr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:34:01.200839 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.200303    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4 podName:28adba45-6c1c-4151-a898-94eaf4974f81 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:05.200282278 +0000 UTC m=+9.277185184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-prfv4" (UniqueName: "kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4") pod "network-check-target-292kr" (UID: "28adba45-6c1c-4151-a898-94eaf4974f81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:34:01.430204 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:01.428898    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:01.430204 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.429035    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:01.430204 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:01.429455    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:01.430204 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.429650    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:01.430204 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:01.429708    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:01.430204 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:01.429781    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:02.554492 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:02.554400    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zh9d6" event={"ID":"0403f76e-6189-4e37-bcb0-b8cecd653283","Type":"ContainerStarted","Data":"91ef3d6faa53e38f3b5b29965568234611b87f06bc7c3b4c9c3c89882c4e9986"}
Oct 30 19:34:02.558292 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:02.557284    2586 generic.go:356] "Generic (PLEG): container finished" podID="76e03f428e399a396d4b35504155fa47" containerID="e753c15a7134ecd9bfbf14a88c0f4fb50550451c8f09d0a72778b03de290e984" exitCode=0
Oct 30 19:34:02.558292 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:02.557362    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"76e03f428e399a396d4b35504155fa47","Type":"ContainerDied","Data":"e753c15a7134ecd9bfbf14a88c0f4fb50550451c8f09d0a72778b03de290e984"}
Oct 30 19:34:02.576848 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:02.576060    2586 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="aedb59cb97ab4dc9d8c44e2befce090534019553fd7bac52954ff1541159362e" exitCode=0
Oct 30 19:34:02.576848 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:02.576105    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerDied","Data":"aedb59cb97ab4dc9d8c44e2befce090534019553fd7bac52954ff1541159362e"}
Oct 30 19:34:03.429502 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:03.428665    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:03.429502 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:03.428798    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:03.429502 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:03.429202    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:03.429502 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:03.429307    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:03.429502 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:03.429371    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:03.429502 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:03.429442    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:03.584213 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:03.584127    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"76e03f428e399a396d4b35504155fa47","Type":"ContainerStarted","Data":"3078d5d78c6fcc23a3bce6885d49596d14d2ce75e82d2462cdd7072220a75028"}
Oct 30 19:34:03.589657 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:03.589581    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mb5hn" event={"ID":"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5","Type":"ContainerStarted","Data":"27d9923d1b775d019e662fdab6f1e1cd954997ab0526841902a56bec5543ce0c"}
Oct 30 19:34:03.674728 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:03.674038    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" podStartSLOduration=6.674017358 podStartE2EDuration="6.674017358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:03.630348054 +0000 UTC m=+7.707250965" watchObservedRunningTime="2026-10-30 19:34:03.674017358 +0000 UTC m=+7.750920271"
Oct 30 19:34:03.674728 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:03.674481    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mb5hn" podStartSLOduration=2.149969955 podStartE2EDuration="7.674456352s" totalImagesPullingTime="5.524486397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:03.673582738 +0000 UTC m=+7.750485650" watchObservedRunningTime="2026-10-30 19:34:03.674456352 +0000 UTC m=+7.751359263"
Oct 30 19:34:04.476747 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:04.472583    2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:04.476747 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:04.473311    2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:04.518094 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:04.517332    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zh9d6" podStartSLOduration=3.97189579 podStartE2EDuration="8.51731915s" totalImagesPullingTime="4.54542336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:03.723698255 +0000 UTC m=+7.800601166" watchObservedRunningTime="2026-10-30 19:34:04.51731915 +0000 UTC m=+8.594222060"
Oct 30 19:34:04.599552 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:04.599512    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" event={"ID":"d8c3ec59a3a8e5471fc2a192c7750dd6","Type":"ContainerStarted","Data":"80e316af4683e6dde93b73a032157a4c948d65ff9e5ec40524fadab1a3fbe5e6"}
Oct 30 19:34:04.603891 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:04.603858    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d" event={"ID":"c37d572e-f147-442e-8b80-18a23ed12528","Type":"ContainerStarted","Data":"c6ddb1ee371dd9321eced5ecc10ceb6ce38b0e6c2c17117d720cf62265dcde38"}
Oct 30 19:34:04.641968 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:04.641910    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" podStartSLOduration=7.641895555 podStartE2EDuration="7.641895555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:04.641665394 +0000 UTC m=+8.718568304" watchObservedRunningTime="2026-10-30 19:34:04.641895555 +0000 UTC m=+8.718798600"
Oct 30 19:34:05.137336 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:05.137294    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:05.137529 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:05.137361    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:05.137529 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.137497    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:34:05.137634 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.137547    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:13.13753411 +0000 UTC m=+17.214436998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:34:05.137881 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.137863    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:34:05.137967 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.137906    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret podName:c16bcfc9-20dd-4a12-ab7b-79dcbcb12557 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:13.137895454 +0000 UTC m=+17.214798341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret") pod "global-pull-secret-syncer-htpxk" (UID: "c16bcfc9-20dd-4a12-ab7b-79dcbcb12557") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:34:05.237979 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:05.237940    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:05.238187 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.238155    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:34:05.238187 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.238180    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:34:05.238307 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.238192    2586 projected.go:196] Error preparing data for projected volume kube-api-access-prfv4 for pod openshift-network-diagnostics/network-check-target-292kr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:34:05.238307 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.238249    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4 podName:28adba45-6c1c-4151-a898-94eaf4974f81 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:13.238232175 +0000 UTC m=+17.315135078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-prfv4" (UniqueName: "kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4") pod "network-check-target-292kr" (UID: "28adba45-6c1c-4151-a898-94eaf4974f81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:34:05.429187 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:05.429153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:05.429361 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.429299    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:05.430067 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:05.429765    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:05.430067 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.429883    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:05.430067 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:05.429949    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:05.430067 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:05.430021    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:05.606807 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:05.605828    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:34:06.014968 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.014949    2586 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:34:06.429210 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.429106    2586 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:34:06.014966099Z","UUID":"2d092119-9668-4bbf-b4f7-262c77ffd371","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:34:06.431552 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.431532    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:34:06.431638 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.431564    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:34:06.613389 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.613349    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xs9fh" event={"ID":"9d96c176-1899-4867-8855-09bfb02134e2","Type":"ContainerStarted","Data":"096408e1ad7c5587fc8ec617dcd7a9446f8803aa436b6af1196632b49a173522"}
Oct 30 19:34:06.618581 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.618548    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d" event={"ID":"c37d572e-f147-442e-8b80-18a23ed12528","Type":"ContainerStarted","Data":"1e7d7712979d1c8475a23402b4bdfafe79ca719381be761cd4f556356959d186"}
Oct 30 19:34:06.620500 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.620445    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qmwpk" event={"ID":"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679","Type":"ContainerStarted","Data":"0b10f31001adba0448d997e81191f108f8b1adb6111469f2332c0621ccaba329"}
Oct 30 19:34:06.702640 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:06.701787    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xs9fh" podStartSLOduration=2.426423919 podStartE2EDuration="10.701768054s" totalImagesPullingTime="8.275344135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:06.660149791 +0000 UTC m=+10.737052700" watchObservedRunningTime="2026-10-30 19:34:06.701768054 +0000 UTC m=+10.778670966"
Oct 30 19:34:07.429373 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.429122    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:07.429373 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.429366    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:07.429594 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.429327    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:07.429594 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:07.429498    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:07.429881 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:07.429849    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:07.430004 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:07.429945    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:07.535224 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.535141    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:07.535386 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.535269    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:34:07.536263 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.536241    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:07.587688 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.587632    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qmwpk" podStartSLOduration=3.072349418 podStartE2EDuration="11.587616966s" totalImagesPullingTime="8.515267548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:06.702580465 +0000 UTC m=+10.779483375" watchObservedRunningTime="2026-10-30 19:34:07.587616966 +0000 UTC m=+11.664519876"
Oct 30 19:34:07.625676 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.625642    2586 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="a2195f83d58646051ea5dc073a0f976cc20b1f4ae2105e4e71df43a8e1ad5225" exitCode=0
Oct 30 19:34:07.626127 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.626010    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerDied","Data":"a2195f83d58646051ea5dc073a0f976cc20b1f4ae2105e4e71df43a8e1ad5225"}
Oct 30 19:34:07.629405 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.629373    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d" event={"ID":"c37d572e-f147-442e-8b80-18a23ed12528","Type":"ContainerStarted","Data":"00a8cd5f307c7e34b68713a6ac08107388029354b1e434ad0c8c17e8c2fc2176"}
Oct 30 19:34:07.720948 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:07.720890    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d" podStartSLOduration=2.956004892 podStartE2EDuration="11.720872204s" totalImagesPullingTime="8.764867312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:07.720824598 +0000 UTC m=+11.797727508" watchObservedRunningTime="2026-10-30 19:34:07.720872204 +0000 UTC m=+11.797775115"
Oct 30 19:34:08.635382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:08.635310    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hsdk9" event={"ID":"b290737d-48ee-4120-a1c8-46917d8f982c","Type":"ContainerStarted","Data":"93e04fe2a6eba3ff39e5708c094803138b4dd9dcf884b26788963f7adea2ea0d"}
Oct 30 19:34:08.680346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:08.680293    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hsdk9" podStartSLOduration=4.006798235 podStartE2EDuration="12.680275321s" totalImagesPullingTime="8.673477086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:08.679683935 +0000 UTC m=+12.756586846" watchObservedRunningTime="2026-10-30 19:34:08.680275321 +0000 UTC m=+12.757178232"
Oct 30 19:34:09.429553 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:09.429512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:09.429553 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:09.429551    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:09.429782 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:09.429560    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:09.429782 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:09.429655    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:09.429782 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:09.429770    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:09.429880 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:09.429834    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:09.639545 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:09.639507    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpc8d" event={"ID":"5f9691d9-c7cd-4b98-8003-39cc09d90a02","Type":"ContainerStarted","Data":"5e3c8f52fa7982a888e06ae3cbbd76a2f71893d1c2edad28f6389b5f785aa083"}
Oct 30 19:34:09.641894 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:09.641863    2586 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="9b58f8a81a3e28e0f64e882a5066d82f8c465090a3e91bbc49edde0f572f459a" exitCode=0
Oct 30 19:34:09.642030 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:09.641949    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerDied","Data":"9b58f8a81a3e28e0f64e882a5066d82f8c465090a3e91bbc49edde0f572f459a"}
Oct 30 19:34:09.686320 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:09.686244    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rpc8d" podStartSLOduration=2.780355089 podStartE2EDuration="13.686229454s" totalImagesPullingTime="10.905874365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:09.686099646 +0000 UTC m=+13.763002556" watchObservedRunningTime="2026-10-30 19:34:09.686229454 +0000 UTC m=+13.763132364"
Oct 30 19:34:10.646970 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:10.646938    2586 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="e07783f8fd5db63dbc099f2fc0ce73a8a17b2a9e0dc2149d3bc4e8119e2ee1ee" exitCode=0
Oct 30 19:34:10.647547 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:10.647023    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerDied","Data":"e07783f8fd5db63dbc099f2fc0ce73a8a17b2a9e0dc2149d3bc4e8119e2ee1ee"}
Oct 30 19:34:11.429589 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:11.429336    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:11.429768 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:11.429375    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:11.429768 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:11.429681    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:11.429768 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:11.429746    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:11.429768 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:11.429383    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:11.429960 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:11.429818    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:12.654388 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:12.654317    2586 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="669d74b6fc5b3fc014887348f243bbb36004342397fa8df83817e673a093bd6f" exitCode=0
Oct 30 19:34:12.654388 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:12.654361    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerDied","Data":"669d74b6fc5b3fc014887348f243bbb36004342397fa8df83817e673a093bd6f"}
Oct 30 19:34:13.204098 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.203943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:13.204271 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.204127    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:13.204271 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.204070    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:34:13.204271 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.204207    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:34:13.204271 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.204235    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret podName:c16bcfc9-20dd-4a12-ab7b-79dcbcb12557 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:29.204213698 +0000 UTC m=+33.281116600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret") pod "global-pull-secret-syncer-htpxk" (UID: "c16bcfc9-20dd-4a12-ab7b-79dcbcb12557") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:34:13.204271 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.204256    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:29.204245655 +0000 UTC m=+33.281148542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:34:13.305238 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.305215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:13.305354 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.305342    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:34:13.305386 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.305359    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:34:13.305386 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.305368    2586 projected.go:196] Error preparing data for projected volume kube-api-access-prfv4 for pod openshift-network-diagnostics/network-check-target-292kr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:34:13.305444 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.305410    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4 podName:28adba45-6c1c-4151-a898-94eaf4974f81 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:29.305395172 +0000 UTC m=+33.382298060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-prfv4" (UniqueName: "kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4") pod "network-check-target-292kr" (UID: "28adba45-6c1c-4151-a898-94eaf4974f81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:34:13.429691 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.429652    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:13.429848 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.429726    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:13.429848 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.429816    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:13.429848 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.429831    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:13.429983 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.429909    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:13.429983 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:13.429961    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:13.659821 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.659733    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"383f9adc36bdf30567520f22db28a20dde0c152218268f779c03ee9eee5c5031"}
Oct 30 19:34:13.659821 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.659814    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"507c6fda7f75b4df0425f0e1e4fcde4bcc7d3761df7e407f88b43c675a749f59"}
Oct 30 19:34:13.660387 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.659850    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"7b80368d479bdafbd6094f439633fe7364454cdf20895a10ecfd7ea8fa20407b"}
Oct 30 19:34:13.660387 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.659887    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"653bee947ee7592f55b52a769dc88fec7a1b898ed52e9b08515bbb2c4fdcfb72"}
Oct 30 19:34:13.660387 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.659917    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"361e648b54a98314afe558bf4b5477690d39528313bdad725292e1541e48757c"}
Oct 30 19:34:13.660387 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.659962    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"9b197267fedcdb87a427982621ca48c8206c448a918e71a4d75212a926789490"}
Oct 30 19:34:13.662661 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.662639    2586 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="2fa816b819cb1234b67705e0443b59a2358297c5c2f910814ef5defd79b5dc0f" exitCode=0
Oct 30 19:34:13.662751 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:13.662672    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerDied","Data":"2fa816b819cb1234b67705e0443b59a2358297c5c2f910814ef5defd79b5dc0f"}
Oct 30 19:34:14.667589 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:14.667551    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" event={"ID":"51c72a9e-1e40-4f4d-ae0f-b42a4923c634","Type":"ContainerStarted","Data":"97a4f74259f16227c3b0e6928ff8c1f6878528d330276b231a76b83a05b7a881"}
Oct 30 19:34:14.745646 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:14.745602    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9mz4v" podStartSLOduration=6.831024896 podStartE2EDuration="17.745589341s" totalImagesPullingTime="10.914564445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:14.731682564 +0000 UTC m=+18.808585474" watchObservedRunningTime="2026-10-30 19:34:14.745589341 +0000 UTC m=+18.822492251"
Oct 30 19:34:15.428947 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:15.428922    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:15.429108 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:15.429090    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:15.429145 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:15.429109    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:15.429214 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:15.429196    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:15.429261 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:15.429251    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:15.429317 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:15.429303    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:15.672301 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:15.672270    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"65a868bd9fca31365ef6c4333c86ad6319beeac2f353440ea80a20cab13222d2"}
Oct 30 19:34:17.429544 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:17.429512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:17.429994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:17.429512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:17.429994 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:17.429635    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:17.429994 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:17.429519    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:17.429994 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:17.429697    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:17.429994 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:17.429803    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:17.680167 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:17.679909    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" event={"ID":"734659d4-c70c-40be-8232-3f2d844704c1","Type":"ContainerStarted","Data":"318f5aed493d530cf53aa7c0e83094cca5c20e5eb9d4d7a9f98b73b249024e9f"}
Oct 30 19:34:17.680307 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:17.680225    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:17.695667 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:17.695644    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:17.774144 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:17.774096    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc" podStartSLOduration=6.535445233 podStartE2EDuration="21.774084178s" totalImagesPullingTime="15.238638945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:17.759678033 +0000 UTC m=+21.836580942" watchObservedRunningTime="2026-10-30 19:34:17.774084178 +0000 UTC m=+21.850987088"
Oct 30 19:34:18.682323 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:18.682291    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:34:18.682918 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:18.682769    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:18.704160 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:18.704131    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:19.429310 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.429277    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:19.429310 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.429300    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:19.429602 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.429304    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:19.429602 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:19.429407    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:19.429602 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:19.429561    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:19.429755 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:19.429688    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:19.563651 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.563569    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-htpxk"]
Oct 30 19:34:19.567720 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.567685    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-292kr"]
Oct 30 19:34:19.567849 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.567744    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ktz22"]
Oct 30 19:34:19.683969 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.683945    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:19.684374 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.684027    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:34:19.684374 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.683945    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:19.684374 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:19.683945    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:19.684374 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:19.684078    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:19.684374 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:19.684120    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:19.684374 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:19.684191    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:20.686579 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:20.686551    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:34:21.429632 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:21.429598    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:21.429632 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:21.429634    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:21.429832 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:21.429640    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:21.429832 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:21.429718    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:21.429832 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:21.429812    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:21.429973 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:21.429901    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:23.429573 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:23.429544    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:23.429957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:23.429544    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:23.429957 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:23.429658    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:23.429957 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:23.429544    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:23.429957 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:23.429716    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:23.429957 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:23.429778    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:25.428701 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.428672    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:25.429137 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.428676    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:25.429137 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:25.428774    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htpxk" podUID="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557"
Oct 30 19:34:25.429137 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.428683    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:25.429137 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:25.428855    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktz22" podUID="4e4bc0c6-9f96-4687-a645-5b727436c653"
Oct 30 19:34:25.429137 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:25.428931    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-292kr" podUID="28adba45-6c1c-4151-a898-94eaf4974f81"
Oct 30 19:34:25.686255 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.686194    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeReady"
Oct 30 19:34:25.686385 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.686308    2586 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:34:25.744606 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.744582    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5wrbn"]
Oct 30 19:34:25.750691 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.750673    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-45g2c"]
Oct 30 19:34:25.750808 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.750796    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.752922 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.752773    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:25.768941 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.768921    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5wrbn"]
Oct 30 19:34:25.771051 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.771010    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:34:25.771169 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.771063    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:34:25.771359 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.771339    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mtjn\""
Oct 30 19:34:25.773091 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.773073    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45g2c"]
Oct 30 19:34:25.773683 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.773667    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:34:25.782282 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.782264    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:34:25.782358 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.782346    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:34:25.784736 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.784723    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk4qf\""
Oct 30 19:34:25.895460 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.895432    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.895610 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.895514    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/7aac5a86-e546-4160-8867-5610087a1b42-kube-api-access-cxrtk\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.895610 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.895544    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aac5a86-e546-4160-8867-5610087a1b42-config-volume\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.895610 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.895560    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aac5a86-e546-4160-8867-5610087a1b42-tmp-dir\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.895610 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.895584    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlsb\" (UniqueName: \"kubernetes.io/projected/cd498b84-3236-409b-8e94-e8ea178171d9-kube-api-access-hnlsb\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:25.895754 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.895625    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd498b84-3236-409b-8e94-e8ea178171d9-cert\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:25.996267 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.996205    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.996376 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:25.996357    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:25.996409 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.996369    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/7aac5a86-e546-4160-8867-5610087a1b42-kube-api-access-cxrtk\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.996442 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:25.996424    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.496404979 +0000 UTC m=+30.573307883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:25.996508 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.996455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aac5a86-e546-4160-8867-5610087a1b42-config-volume\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.996508 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.996495    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aac5a86-e546-4160-8867-5610087a1b42-tmp-dir\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.996604 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.996524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlsb\" (UniqueName: \"kubernetes.io/projected/cd498b84-3236-409b-8e94-e8ea178171d9-kube-api-access-hnlsb\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:25.996604 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.996551    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd498b84-3236-409b-8e94-e8ea178171d9-cert\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:25.996852 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.996836    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aac5a86-e546-4160-8867-5610087a1b42-tmp-dir\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:25.997088 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:25.997068    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aac5a86-e546-4160-8867-5610087a1b42-config-volume\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:26.000668 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:26.000646    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd498b84-3236-409b-8e94-e8ea178171d9-cert\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:26.022642 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:26.022615    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlsb\" (UniqueName: \"kubernetes.io/projected/cd498b84-3236-409b-8e94-e8ea178171d9-kube-api-access-hnlsb\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:26.022738 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:26.022721    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/7aac5a86-e546-4160-8867-5610087a1b42-kube-api-access-cxrtk\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:26.067922 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:26.067894    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:26.428841 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:26.428814    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45g2c"]
Oct 30 19:34:26.429980 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:34:26.429956    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd498b84_3236_409b_8e94_e8ea178171d9.slice/crio-c1d813cb9fd093431a3df1c31caeebedcf3682b20cdf8569a470901f73c88afb WatchSource:0}: Error finding container c1d813cb9fd093431a3df1c31caeebedcf3682b20cdf8569a470901f73c88afb: Status 404 returned error can't find the container with id c1d813cb9fd093431a3df1c31caeebedcf3682b20cdf8569a470901f73c88afb
Oct 30 19:34:26.501273 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:26.501245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:26.501395 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:26.501345    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:26.501395 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:26.501389    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:27.501378121 +0000 UTC m=+31.578281008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:26.699065 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:26.698977    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45g2c" event={"ID":"cd498b84-3236-409b-8e94-e8ea178171d9","Type":"ContainerStarted","Data":"c1d813cb9fd093431a3df1c31caeebedcf3682b20cdf8569a470901f73c88afb"}
Oct 30 19:34:27.428646 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.428612    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:27.428830 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.428682    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:27.428830 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.428786    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:27.446137 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.446084    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:34:27.446137 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.446105    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:34:27.460156 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.460131    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6fz68\""
Oct 30 19:34:27.460270 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.460195    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:34:27.460270 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.460212    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zv66j\""
Oct 30 19:34:27.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.460283    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:34:27.460382 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.460357    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52dmm\""
Oct 30 19:34:27.508334 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.508307    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:27.508493 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:27.508418    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:27.508566 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:27.508494    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:29.508462335 +0000 UTC m=+33.585365226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:27.682255 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.682177    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:27.682558 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.682377    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:34:27.698730 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:27.698705    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:29.222293 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.222270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:29.222671 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.222310    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:29.222671 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:29.222427    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:34:29.222671 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:29.222487    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:35:01.22246135 +0000 UTC m=+65.299364239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : secret "metrics-daemon-secret" not found
Oct 30 19:34:29.224556 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.224535    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:29.285252 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.285217    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:29.323097 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.323075    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:29.325619 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.325587    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:29.525089 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.525064    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:29.525187 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:29.525180    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:29.525242 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:29.525232    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.525220037 +0000 UTC m=+37.602122925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:29.563887 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.563860    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:29.616968 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.616938    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-htpxk"]
Oct 30 19:34:29.618594 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:34:29.618568    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16bcfc9_20dd_4a12_ab7b_79dcbcb12557.slice/crio-4f5527b3ee17cecdc656ee10c5931b686b7553e1db105ed9627eab06c4a5a7e6 WatchSource:0}: Error finding container 4f5527b3ee17cecdc656ee10c5931b686b7553e1db105ed9627eab06c4a5a7e6: Status 404 returned error can't find the container with id 4f5527b3ee17cecdc656ee10c5931b686b7553e1db105ed9627eab06c4a5a7e6
Oct 30 19:34:29.707403 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.707367    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45g2c" event={"ID":"cd498b84-3236-409b-8e94-e8ea178171d9","Type":"ContainerStarted","Data":"9b300992cbed2956fa1fbf4734b670edae00a87603b74f7255f8de95960a45ba"}
Oct 30 19:34:29.708621 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.708596    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-htpxk" event={"ID":"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557","Type":"ContainerStarted","Data":"4f5527b3ee17cecdc656ee10c5931b686b7553e1db105ed9627eab06c4a5a7e6"}
Oct 30 19:34:29.752361 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.752296    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-45g2c" podStartSLOduration=2.054649903 podStartE2EDuration="4.752279964s" totalImagesPullingTime="2.697630061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:29.751779656 +0000 UTC m=+33.828682567" watchObservedRunningTime="2026-10-30 19:34:29.752279964 +0000 UTC m=+33.829182874"
Oct 30 19:34:29.883913 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:29.883882    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-292kr"]
Oct 30 19:34:29.884192 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:34:29.884169    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28adba45_6c1c_4151_a898_94eaf4974f81.slice/crio-9967c4d42e0c33944033d31c6e2f04035710583e0d7b4c15dfb0c6eb7ac97cd8 WatchSource:0}: Error finding container 9967c4d42e0c33944033d31c6e2f04035710583e0d7b4c15dfb0c6eb7ac97cd8: Status 404 returned error can't find the container with id 9967c4d42e0c33944033d31c6e2f04035710583e0d7b4c15dfb0c6eb7ac97cd8
Oct 30 19:34:30.713135 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.713084    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-292kr" event={"ID":"28adba45-6c1c-4151-a898-94eaf4974f81","Type":"ContainerStarted","Data":"9967c4d42e0c33944033d31c6e2f04035710583e0d7b4c15dfb0c6eb7ac97cd8"}
Oct 30 19:34:30.837409 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.837374    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7kstg"]
Oct 30 19:34:30.859985 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.859959    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:30.863016 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.862938    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kstg"]
Oct 30 19:34:30.885902 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.885876    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-h2cdk-f7vp9"]
Oct 30 19:34:30.897533 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.897510    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:30.912298 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.912234    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h2cdk-f7vp9"]
Oct 30 19:34:30.937422 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.937389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a037daa7-43b1-4565-8bf5-debb79042d0c-crio-socket\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:30.937643 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.937451    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a037daa7-43b1-4565-8bf5-debb79042d0c-data-volume\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:30.937643 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.937517    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:30.937643 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.937538    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7tz\" (UniqueName: \"kubernetes.io/projected/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-api-access-lc7tz\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:30.937643 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.937565    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:30.965757 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.965549    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:34:30.965757 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.965577    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hmkn8\""
Oct 30 19:34:30.965757 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.965623    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:34:30.965757 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.965624    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:34:30.965757 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.965643    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hgf7z\""
Oct 30 19:34:30.965757 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.965694    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:34:30.965757 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.965740    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:34:30.994346 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:30.994305    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:34:31.038573 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038542    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a037daa7-43b1-4565-8bf5-debb79042d0c-crio-socket\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.038739 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038585    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.038739 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038634    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a037daa7-43b1-4565-8bf5-debb79042d0c-data-volume\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.038739 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038667    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.038739 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038709    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.038739 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038735    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.038982 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038758    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7tz\" (UniqueName: \"kubernetes.io/projected/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-api-access-lc7tz\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.038982 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038776    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a037daa7-43b1-4565-8bf5-debb79042d0c-crio-socket\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.038982 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038790    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhjrs\" (UniqueName: \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.038982 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.038881    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.039152 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.038989    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:31.039152 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.039046    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:31.539027692 +0000 UTC m=+35.615930598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:31.039152 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.039073    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a037daa7-43b1-4565-8bf5-debb79042d0c-data-volume\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.039152 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.039145    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:34:31.039339 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.039207    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:31.539188775 +0000 UTC m=+35.616091683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : configmap "kube-rbac-proxy" not found
Oct 30 19:34:31.062132 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.062104    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7tz\" (UniqueName: \"kubernetes.io/projected/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-api-access-lc7tz\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.139784 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.139750    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.139941 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.139819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.139941 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.139866    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.139941 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.139905    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhjrs\" (UniqueName: \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.140173 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.140152    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.140725 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.140699    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.140990 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.140971    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.167163 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.167131    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhjrs\" (UniqueName: \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.231033 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.230953    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:31.543458 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.543370    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.543458 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.543422    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:31.543781 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.543526    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:34:31.543781 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.543584    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:31.543781 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.543602    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:32.54358472 +0000 UTC m=+36.620487609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : configmap "kube-rbac-proxy" not found
Oct 30 19:34:31.543781 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:31.543648    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:32.543626925 +0000 UTC m=+36.620529838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:31.556591 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.556565    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h2cdk-f7vp9"]
Oct 30 19:34:31.558194 ip-10-0-133-26 kubenswrapper[2586]: W1030 19:34:31.558165    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35623b34_ad3c_4170_99a2_76ff61156c80.slice/crio-c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09 WatchSource:0}: Error finding container c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09: Status 404 returned error can't find the container with id c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09
Oct 30 19:34:31.717182 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:31.717144    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h2cdk-f7vp9" event={"ID":"35623b34-ad3c-4170-99a2-76ff61156c80","Type":"ContainerStarted","Data":"c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09"}
Oct 30 19:34:32.031052 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.030206    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"]
Oct 30 19:34:32.061064 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.061032    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"]
Oct 30 19:34:32.061233 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.061160    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"
Oct 30 19:34:32.106046 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.106014    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-khwrx\""
Oct 30 19:34:32.122085 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.122054    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:34:32.122234 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.122192    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:34:32.149874 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.149843    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwl6c\" (UniqueName: \"kubernetes.io/projected/8e4e5602-243d-4855-9350-02d1c05762cf-kube-api-access-mwl6c\") pod \"migrator-66df5d496b-vr4bv\" (UID: \"8e4e5602-243d-4855-9350-02d1c05762cf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"
Oct 30 19:34:32.251227 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.251190    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwl6c\" (UniqueName: \"kubernetes.io/projected/8e4e5602-243d-4855-9350-02d1c05762cf-kube-api-access-mwl6c\") pod \"migrator-66df5d496b-vr4bv\" (UID: \"8e4e5602-243d-4855-9350-02d1c05762cf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"
Oct 30 19:34:32.274219 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.274182    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwl6c\" (UniqueName: \"kubernetes.io/projected/8e4e5602-243d-4855-9350-02d1c05762cf-kube-api-access-mwl6c\") pod \"migrator-66df5d496b-vr4bv\" (UID: \"8e4e5602-243d-4855-9350-02d1c05762cf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"
Oct 30 19:34:32.387301 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.387198    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"
Oct 30 19:34:32.554374 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.554173    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:32.554374 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.554230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:32.554374 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:32.554300    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:32.554374 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:32.554320    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:34:32.554374 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:32.554369    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:34.554353789 +0000 UTC m=+38.631256677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:32.554374 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:32.554386    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:34.554379009 +0000 UTC m=+38.631281897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : configmap "kube-rbac-proxy" not found
Oct 30 19:34:32.706200 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.706162    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"]
Oct 30 19:34:32.722180 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:32.722103    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv" event={"ID":"8e4e5602-243d-4855-9350-02d1c05762cf","Type":"ContainerStarted","Data":"f0786b5ca278c4030659896f954bea884dab2ec315022f5392fdab26784c9614"}
Oct 30 19:34:33.003434 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.003354    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5qphz"]
Oct 30 19:34:33.026962 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.026931    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.027878 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.027371    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5qphz"]
Oct 30 19:34:33.044403 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.044339    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:34:33.044403 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.044339    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:34:33.044593 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.044411    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:34:33.057954 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.057909    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:34:33.058065 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.058005    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7j4sn\""
Oct 30 19:34:33.159816 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.159780    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-key\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.160004 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.159986    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzk4\" (UniqueName: \"kubernetes.io/projected/b00be7b0-0f7e-4254-a128-e2fe6ee92277-kube-api-access-fgzk4\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.160069 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.160032    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.260787 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.260696    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzk4\" (UniqueName: \"kubernetes.io/projected/b00be7b0-0f7e-4254-a128-e2fe6ee92277-kube-api-access-fgzk4\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.260787 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.260769    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.261092 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.260797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-key\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.261631 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.261580    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.263743 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.263703    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-key\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.295796 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.295717    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzk4\" (UniqueName: \"kubernetes.io/projected/b00be7b0-0f7e-4254-a128-e2fe6ee92277-kube-api-access-fgzk4\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.355930 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.355663    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:33.563710 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.563117    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:33.563710 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:33.563248    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:33.563710 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:33.563308    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.563290604 +0000 UTC m=+45.640193495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:33.686215 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.686182    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5qphz"]
Oct 30 19:34:33.726818 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:33.726777    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz" event={"ID":"b00be7b0-0f7e-4254-a128-e2fe6ee92277","Type":"ContainerStarted","Data":"d0bd988ff2ad535bad7a635ff0a44bc6f6614aa9ead821d09315a30a2f50a194"}
Oct 30 19:34:34.572508 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:34.572459    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:34.572679 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:34.572522    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:34.572679 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:34.572620    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:34:34.572679 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:34.572651    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:34.572809 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:34.572684    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.572668318 +0000 UTC m=+42.649571207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : configmap "kube-rbac-proxy" not found
Oct 30 19:34:34.572809 ip-10-0-133-26 kubenswrapper[2586]: E1030 19:34:34.572703    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.572690182 +0000 UTC m=+42.649593069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:34.731875 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:34.731843    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-292kr" event={"ID":"28adba45-6c1c-4151-a898-94eaf4974f81","Type":"ContainerStarted","Data":"476120e545ba51dc10b191c34bea4d38c9db76b91f3f747b6cd1aa4553944cb8"}
Oct 30 19:34:34.732300 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:34.731997    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:34.775312 ip-10-0-133-26 kubenswrapper[2586]: I1030 19:34:34.774140    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-292kr" podStartSLOduration=34.334885128 podStartE2EDuration="38.774127994s" totalImagesPullingTime="4.439242866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:34.77411952 +0000 UTC m=+38.851022431" watchObservedRunningTime="2026-10-30 19:34:34.774127994 +0000 UTC m=+38.851030904"
Oct 30 19:34:35.420012 ip-10-0-133-26 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:34:35.435009 ip-10-0-133-26 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:34:35.435347 ip-10-0-133-26 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:34:35.435408 ip-10-0-133-26 systemd[1]: kubelet.service: Consumed 2.337s CPU time.
Oct 30 19:34:35.447025 ip-10-0-133-26 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:34:35.529288 ip-10-0-133-26 kubenswrapper[4519]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:34:35.529288 ip-10-0-133-26 kubenswrapper[4519]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:34:35.529288 ip-10-0-133-26 kubenswrapper[4519]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:34:35.529288 ip-10-0-133-26 kubenswrapper[4519]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:34:35.531435 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531355    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.531435 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531434    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.531435 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531438    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531442    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531446    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531449    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531452    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531455    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531458    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531460    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531475    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531478    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531481    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531484    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531487    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531489    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531493    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531496    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531499    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531501    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531504    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531507    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.531553 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531510    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531512    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531515    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531518    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531521    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531528    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531531    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531534    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531538    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531543    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531546    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531549    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531552    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531556    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531559    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531562    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531565    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531575    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531579    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.532057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531582    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531584    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531587    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531590    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531594    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531597    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531600    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531603    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531605    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531608    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531612    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531615    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531618    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531621    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531623    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531626    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531629    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531632    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531634    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.532541 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531637    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531640    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531642    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531645    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531648    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531650    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531653    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531656    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531659    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531661    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531664    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531668    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531673    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531676    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531679    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531682    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531684    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531687    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531690    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.533013 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531693    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531695    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531698    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531701    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531704    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531707    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531710    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531713    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531715    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531718    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531721    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531724    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531726    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531729    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531732    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531735    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531738    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531740    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531744    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531746    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.533500 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531749    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531752    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531754    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531757    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531760    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531763    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.531766    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532545    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532553    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532556    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532560    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532563    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532566    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532569    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532572    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532574    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532577    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532580    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532583    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532586    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.534001 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532589    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532591    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532594    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532597    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532599    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532602    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532605    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532608    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532610    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532615    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532618    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532620    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532623    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532626    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532628    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532632    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532634    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532637    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532640    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532643    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.534506 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532645    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532648    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532651    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532654    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532657    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532660    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532663    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532665    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532668    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532671    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532674    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532677    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532680    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532683    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532686    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532689    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532691    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532696    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532700    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.535076 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532703    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532706    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532709    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532712    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532724    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532727    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532730    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532732    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532737    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532741    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532744    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532747    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532749    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532752    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532755    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532758    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532761    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532763    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532766    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.535577 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532770    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532772    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532775    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532778    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532781    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532784    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532786    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532789    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532792    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532795    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532798    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532800    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532803    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532805    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532808    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532811    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532814    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532816    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532819    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532822    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.536048 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532824    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532827    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532830    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532833    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532835    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532838    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532841    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532843    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532846    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532848    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532854    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532857    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532860    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532863    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.532866    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533388    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533394    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533398    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533401    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.536552 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533404    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533407    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533410    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533413    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533415    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533418    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533421    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533424    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533426    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533429    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533432    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533434    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533437    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533440    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533442    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533445    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533448    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533450    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533453    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.537032 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533455    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533458    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533461    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533478    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533483    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533488    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533491    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533501    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533504    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533507    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533510    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533512    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533515    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533518    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533521    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533523    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533526    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533529    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533531    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.537527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533534    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533537    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533539    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533543    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533547    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533551    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533553    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533556    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533559    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533563    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533566    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533569    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533572    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533575    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533577    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533580    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533582    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533585    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533588    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.537994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533591    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533594    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533597    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533599    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533602    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533605    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533608    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533610    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533613    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533616    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533619    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533621    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533624    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533627    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533629    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533632    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533634    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533637    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533640    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533642    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.538478 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533645    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533648    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533651    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533654    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533656    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533659    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533662    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533665    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533667    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533670    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533673    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533676    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533679    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533681    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533684    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533687    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533690    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533693    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533696    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533698    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.539021 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533701    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533704    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533706    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533709    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.533712    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533806    4519 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533820    4519 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533827    4519 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533831    4519 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533835    4519 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533839    4519 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533843    4519 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533851    4519 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533855    4519 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533858    4519 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533861    4519 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533865    4519 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533868    4519 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533871    4519 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533874    4519 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533877    4519 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533880    4519 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533883    4519 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:34:35.539554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533888    4519 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533891    4519 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533894    4519 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533897    4519 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533900    4519 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533905    4519 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533908    4519 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533911    4519 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533915    4519 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533918    4519 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533921    4519 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533924    4519 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533927    4519 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533930    4519 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533934    4519 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533937    4519 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533940    4519 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533943    4519 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533946    4519 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533949    4519 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533954    4519 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533958    4519 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533960    4519 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533964    4519 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:34:35.540242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533967    4519 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533971    4519 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533974    4519 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533977    4519 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533980    4519 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533984    4519 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533990    4519 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533993    4519 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533996    4519 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.533999    4519 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534002    4519 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534006    4519 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534010    4519 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534013    4519 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534017    4519 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534020    4519 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534023    4519 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534026    4519 flags.go:64] FLAG: --help="false"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534032    4519 flags.go:64] FLAG: --hostname-override="ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534035    4519 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534038    4519 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534041    4519 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534045    4519 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:34:35.541121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534049    4519 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534052    4519 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534055    4519 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534058    4519 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534061    4519 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534064    4519 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534067    4519 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534070    4519 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534073    4519 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534076    4519 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534079    4519 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534082    4519 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534085    4519 flags.go:64] FLAG: --lock-file=""
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534088    4519 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534091    4519 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534096    4519 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534102    4519 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534105    4519 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534108    4519 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534111    4519 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534114    4519 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534117    4519 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534120    4519 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534124    4519 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534129    4519 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:34:35.541711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534132    4519 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534136    4519 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534140    4519 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534144    4519 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534147    4519 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534150    4519 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534153    4519 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534156    4519 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534158    4519 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534166    4519 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534169    4519 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534172    4519 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534175    4519 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534178    4519 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534181    4519 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534184    4519 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534187    4519 flags.go:64] FLAG: --port="10250"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534190    4519 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534193    4519 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b79a218db378e734"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534196    4519 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534199    4519 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534203    4519 flags.go:64] FLAG: --register-node="true"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534207    4519 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534211    4519 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534214    4519 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534217    4519 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:34:35.542308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534220    4519 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534224    4519 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534228    4519 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534231    4519 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534234    4519 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534237    4519 flags.go:64] FLAG: --runonce="false"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534241    4519 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534245    4519 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534252    4519 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534255    4519 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534258    4519 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534261    4519 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534264    4519 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534268    4519 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534270    4519 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534273    4519 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534277    4519 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534279    4519 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534283    4519 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534286    4519 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534289    4519 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534294    4519 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534297    4519 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534300    4519 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534304    4519 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:34:35.543325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534307    4519 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534310    4519 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534313    4519 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534318    4519 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534321    4519 flags.go:64] FLAG: --v="2"
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534326    4519 flags.go:64] FLAG: --version="false"
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534330    4519 flags.go:64] FLAG: --vmodule=""
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534334    4519 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534337    4519 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534478    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534482    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534485    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534488    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534491    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534494    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534503    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534506    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534509    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534512    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534515    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534518    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.544267 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534520    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534523    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534526    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534529    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534532    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534534    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534537    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534540    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534543    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534545    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534548    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534551    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534553    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534556    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534561    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534564    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534567    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534569    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534572    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534575    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.544813 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534578    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534580    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534583    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534586    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534589    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534593    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534604    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534608    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534611    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534615    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534618    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534621    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534623    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534626    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534629    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534631    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534634    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534637    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534640    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534643    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.545339 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534645    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534648    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534651    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534654    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534656    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534659    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534663    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534666    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534668    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534671    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534675    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534678    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534682    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534685    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534688    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534690    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534693    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534696    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534700    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.545860 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534708    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534712    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534714    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534717    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534720    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534722    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534726    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534728    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534731    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534734    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534737    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534740    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534742    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534745    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534748    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534750    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534753    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534756    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534758    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534762    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.546370 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534765    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534768    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534771    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534773    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534776    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534779    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534781    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534784    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534787    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534790    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534792    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534796    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534799    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534807    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534810    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.534835    4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:34:35.547123 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534871    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534875    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534878    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534881    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534884    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534887    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534889    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534892    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534895    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534898    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534900    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534903    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534906    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534908    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534911    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534916    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534918    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534932    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534934    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.547564 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534937    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534940    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534943    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534945    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534948    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534951    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534954    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534956    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534959    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534963    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534966    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534969    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534972    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534975    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534978    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534980    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534983    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534986    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534988    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534991    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.548041 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534994    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534996    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.534999    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535003    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535007    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535010    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535013    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535016    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535020    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535023    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535025    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535028    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535031    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535033    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535036    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535039    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535042    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535044    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535047    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.548620 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535050    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535053    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535057    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535060    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535063    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535066    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535069    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535071    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535074    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535078    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535082    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535086    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535089    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535092    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535095    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535097    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535100    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535103    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535105    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.549190 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535108    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535111    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535114    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535117    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535120    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535123    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535126    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535129    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535131    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535134    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535137    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535139    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535142    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535145    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535147    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535151    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535154    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535157    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535160    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535169    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.563882 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535172    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535175    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535177    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535180    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535183    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535185    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535189    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535192    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.535195    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.539133    4519 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.539151    4519 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.539165    4519 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.539169    4519 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539228    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539233    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539237    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539240    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539243    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539246    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539249    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.564418 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539252    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539255    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539258    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539261    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539264    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539266    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539270    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539273    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539276    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539279    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539282    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539284    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539287    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539290    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539293    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539295    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539298    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539301    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539304    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.564978 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539306    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539309    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539312    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539321    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539324    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539327    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539329    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539332    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539335    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539338    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539340    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539343    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539346    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539349    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539352    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539354    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539357    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539360    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539363    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539366    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.565510 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539369    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539371    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539374    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539377    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539379    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539382    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539384    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539387    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539391    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539395    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539398    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539402    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539406    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539408    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539411    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539414    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539423    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539426    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539429    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.566043 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539432    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539435    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539437    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539440    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539443    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539446    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539448    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539451    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539454    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539456    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539459    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539462    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539481    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539485    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539489    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539492    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539495    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539498    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539501    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539504    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.566570 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539506    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539509    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539512    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539515    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539517    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539520    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539523    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539526    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539528    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539531    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539542    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539545    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539548    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539552    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539556    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539560    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539563    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539571    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539576    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.567108 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539579    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539582    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.539610    4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539790    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539795    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539798    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539801    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539804    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539807    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539810    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539812    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539815    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539818    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539821    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539823    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539826    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.567628 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539829    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539831    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539834    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539837    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539839    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539842    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539845    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539848    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539856    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539859    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539862    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539865    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539868    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539870    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539873    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539876    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539879    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539881    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539884    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.568031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539887    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539890    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539892    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539895    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539898    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539901    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539903    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539906    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539909    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539912    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539914    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539917    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539919    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539922    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539925    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539928    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539930    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539933    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539935    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.568527 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539938    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539941    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539944    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539953    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539956    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539959    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539961    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539964    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539967    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539969    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539973    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539978    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539981    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539983    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539986    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539989    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539991    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539994    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.539997    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.568996 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540000    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540002    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540005    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540008    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540010    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540013    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540021    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540024    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540027    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540032    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540035    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540038    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540040    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540043    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540046    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540048    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540051    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540059    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540062    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540065    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.569460 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540067    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540070    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540073    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540076    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540079    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540082    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540085    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540087    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540090    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540094    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540098    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540101    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540104    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540107    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540110    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540112    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.570018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.540136    4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540173    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540177    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540181    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540184    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540188    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540191    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540194    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540197    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540199    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540202    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540205    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540208    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540210    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540218    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540221    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540224    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540227    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540230    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540232    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:34:35.570411 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540235    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540240    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540243    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540246    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540249    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540252    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540255    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540258    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540260    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540263    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540266    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540269    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540271    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540274    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540277    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540280    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540282    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540286    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540289    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:34:35.570927 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540292    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540295    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540297    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540300    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540303    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540305    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540308    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540311    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540319    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540322    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540325    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540327    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540330    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540333    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540336    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540339    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540342    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540344    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540347    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:34:35.571416 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540352    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540355    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540359    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540361    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540364    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540367    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540369    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540372    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540375    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540377    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540380    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540385    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540388    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540391    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540394    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540396    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540399    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540402    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540404    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:34:35.571899 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540407    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540410    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540412    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540421    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540424    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540426    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540429    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540432    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540434    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540437    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540440    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540442    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540445    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540448    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540450    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540453    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540456    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540459    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540461    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540487    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:34:35.572361 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540492    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540496    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540498    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540501    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540505    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540508    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540511    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540513    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540516    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:35.540519    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.540580    4519 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.541587    4519 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.541637    4519 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.541830    4519 server.go:1014] "Starting client certificate rotation"
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.541978    4519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.542046    4519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:28:56 +0000 UTC" deadline="2028-08-04 11:29:49.321527224 +0000 UTC"
Oct 30 19:34:35.572884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.542113    4519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15447h55m13.77941629s"
Oct 30 19:34:35.573275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.542418    4519 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:34:35.573275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.545079    4519 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:34:35.573275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.546674    4519 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:34:35.573275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.548279    4519 log.go:25] "Validated CRI v1 image API"
Oct 30 19:34:35.573275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.549082    4519 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:34:35.573275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.550689    4519 fs.go:135] Filesystem UUIDs: map[0c3270c1-f715-4db1-96b9-cc847717548f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ef613b7a-5eea-4767-aa65-e978e9b86cd9:/dev/nvme0n1p3]
Oct 30 19:34:35.573481 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.550711    4519 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/072fe2ae46557f05ebf9b841780c4d07797e4fab742e614ad7a3a18ed7ebeec8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/072fe2ae46557f05ebf9b841780c4d07797e4fab742e614ad7a3a18ed7ebeec8/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ea660b33bb399f60b597aa0552a5287568c5ea36bcee75969e142d6a5297dfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ea660b33bb399f60b597aa0552a5287568c5ea36bcee75969e142d6a5297dfa/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f5527b3ee17cecdc656ee10c5931b686b7553e1db105ed9627eab06c4a5a7e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f5527b3ee17cecdc656ee10c5931b686b7553e1db105ed9627eab06c4a5a7e6/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f77ce3f3d68a73ad1c89fa3da4f05cf08e65255b69896c984c2eef9c68187d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f77ce3f3d68a73ad1c89fa3da4f05cf08e65255b69896c984c2eef9c68187d3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ae66028480815792c693695be196914e95bdeacc4913ce4470cfa5013875106/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ae66028480815792c693695be196914e95bdeacc4913ce4470cfa5013875106/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9967c4d42e0c33944033d31c6e2f04035710583e0d7b4c15dfb0c6eb7ac97cd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9967c4d42e0c33944033d31c6e2f04035710583e0d7b4c15dfb0c6eb7ac97cd8/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af9b796f616f8513c134b681bfeea6e16769b0ed5e37422fb8c0e8be8374b279/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af9b796f616f8513c134b681bfeea6e16769b0ed5e37422fb8c0e8be8374b279/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1d813cb9fd093431a3df1c31caeebedcf3682b20cdf8569a470901f73c88afb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1d813cb9fd093431a3df1c31caeebedcf3682b20cdf8569a470901f73c88afb/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c620f89f08c9530f2397517431a8ee3ec99ce67a904cada33f82c575f2cddf2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c620f89f08c9530f2397517431a8ee3ec99ce67a904cada33f82c575f2cddf2f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0bd988ff2ad535bad7a635ff0a44bc6f6614aa9ead821d09315a30a2f50a194/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0bd988ff2ad535bad7a635ff0a44bc6f6614aa9ead821d09315a30a2f50a194/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2ab193b546d6e627434bc6df3032bb57f82b33beeec27ad45063d9f1c9be0bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2ab193b546d6e627434bc6df3032bb57f82b33beeec27ad45063d9f1c9be0bb/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1f3687bc80427c23e2fdfedf273d0e3d7a88794452e3eeede25087e729ed231/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1f3687bc80427c23e2fdfedf273d0e3d7a88794452e3eeede25087e729ed231/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0786b5ca278c4030659896f954bea884dab2ec315022f5392fdab26784c9614/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0786b5ca278c4030659896f954bea884dab2ec315022f5392fdab26784c9614/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0403f76e-6189-4e37-bcb0-b8cecd653283/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0403f76e-6189-4e37-bcb0-b8cecd653283/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28adba45-6c1c-4151-a898-94eaf4974f81/volumes/kubernetes.io~projected/kube-api-access-prfv4:{mountpoint:/var/lib/kubelet/pods/28adba45-6c1c-4151-a898-94eaf4974f81/volumes/kubernetes.io~projected/kube-api-access-prfv4 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35623b34-ad3c-4170-99a2-76ff61156c80/volumes/kubernetes.io~projected/kube-api-access-zhjrs:{mountpoint:/var/lib/kubelet/pods/35623b34-ad3c-4170-99a2-76ff61156c80/volumes/kubernetes.io~projected/kube-api-access-zhjrs major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4bc0c6-9f96-4687-a645-5b727436c653/volumes/kubernetes.io~projected/kube-api-access-v762g:{mountpoint:/var/lib/kubelet/pods/4e4bc0c6-9f96-4687-a645-5b727436c653/volumes/kubernetes.io~projected/kube-api-access-v762g major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51c72a9e-1e40-4f4d-ae0f-b42a4923c634/volumes/kubernetes.io~projected/kube-api-access-phw2p:{mountpoint:/var/lib/kubelet/pods/51c72a9e-1e40-4f4d-ae0f-b42a4923c634/volumes/kubernetes.io~projected/kube-api-access-phw2p major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679/volumes/kubernetes.io~projected/kube-api-access-ffflj:{mountpoint:/var/lib/kubelet/pods/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679/volumes/kubernetes.io~projected/kube-api-access-ffflj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~projected/kube-api-access-2htz2:{mountpoint:/var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~projected/kube-api-access-2htz2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volumes/kubernetes.io~projected/kube-api-access-lx8r5:{mountpoint:/var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volumes/kubernetes.io~projected/kube-api-access-lx8r5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aac5a86-e546-4160-8867-5610087a1b42/volumes/kubernetes.io~projected/kube-api-access-cxrtk:{mountpoint:/var/lib/kubelet/pods/7aac5a86-e546-4160-8867-5610087a1b42/volumes/kubernetes.io~projected/kube-api-access-cxrtk major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e4e5602-243d-4855-9350-02d1c05762cf/volumes/kubernetes.io~projected/kube-api-access-mwl6c:{mountpoint:/var/lib/kubelet/pods/8e4e5602-243d-4855-9350-02d1c05762cf/volumes/kubernetes.io~projected/kube-api-access-mwl6c major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d96c176-1899-4867-8855-09bfb02134e2/volumes/kubernetes.io~projected/kube-api-access-nwfql:{mountpoint:/var/lib/kubelet/pods/9d96c176-1899-4867-8855-09bfb02134e2/volumes/kubernetes.io~projected/kube-api-access-nwfql major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a037daa7-43b1-4565-8bf5-debb79042d0c/volumes/kubernetes.io~projected/kube-api-access-lc7tz:{mountpoint:/var/lib/kubelet/pods/a037daa7-43b1-4565-8bf5-debb79042d0c/volumes/kubernetes.io~projected/kube-api-access-lc7tz major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5/volumes/kubernetes.io~projected/kube-api-access-ltdch:{mountpoint:/var/lib/kubelet/pods/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5/volumes/kubernetes.io~projected/kube-api-access-ltdch major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b00be7b0-0f7e-4254-a128-e2fe6ee92277/volumes/kubernetes.io~projected/kube-api-access-fgzk4:{mountpoint:/var/lib/kubelet/pods/b00be7b0-0f7e-4254-a128-e2fe6ee92277/volumes/kubernetes.io~projected/kube-api-access-fgzk4 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b00be7b0-0f7e-4254-a128-e2fe6ee92277/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b00be7b0-0f7e-4254-a128-e2fe6ee92277/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b290737d-48ee-4120-a1c8-46917d8f982c/volumes/kubernetes.io~projected/kube-api-access-hmrvv:{mountpoint:/var/lib/kubelet/pods/b290737d-48ee-4120-a1c8-46917d8f982c/volumes/kubernetes.io~projected/kube-api-access-hmrvv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c37d572e-f147-442e-8b80-18a23ed12528/volumes/kubernetes.io~projected/kube-api-access-r9v9m:{mountpoint:/var/lib/kubelet/pods/c37d572e-f147-442e-8b80-18a23ed12528/volumes/kubernetes.io~projected/kube-api-access-r9v9m major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd498b84-3236-409b-8e94-e8ea178171d9/volumes/kubernetes.io~projected/kube-api-access-hnlsb:{mountpoint:/var/lib/kubelet/pods/cd498b84-3236-409b-8e94-e8ea178171d9/volumes/kubernetes.io~projected/kube-api-access-hnlsb major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd498b84-3236-409b-8e94-e8ea178171d9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cd498b84-3236-409b-8e94-e8ea178171d9/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f6751745311a316b431520771545561e248a0eacd414eca3b1e3d420c203e729/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d52236a3d083ede40c3fe5a5a9ba1ff2f4a7efcd07a798e78ca524ed0cb34f2d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ec389758fd9267407fec6d0853786de8daa7a9151ada2f7f2d1e54343b9aa5c3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ccf5a8e0377a1053290b9bbd3c811c4a42b66131c53385b15e2090a78111c7d3/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bb9872586243acded0222f4535b958fcc2b4aae5b85dbc3c79c5fc61d9b30410/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1e6cfb15c33423cdbd6bdc6ed3d7050436aa3adc2d5443ce412cd4b30a123c13/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/60d10bb4e22dc790f678336b51f833ff927d8667605aac203bc94f7ae6371aea/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e4a168a4ff379945c5a53dea3084c100c6615419937a0113e0b1ad9ce2a50bbb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9c771b11d244da33be79a8dc0230e2bc988e44945a3f8060a0cf44b02ba92a21/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/3203f974cc2748944cbf10f7adae7a45bea59f766c9f8e485121e21d2ef626ce/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/d1352d313f10009d6d19d81348424879d2877290811b1a1d8d2c12be32e5c103/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/8258ea180e835bb96357309c0b1fd0706a1aac9788d2cad5b54fa872bef798ad/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/12cfddc4e8c08a82664d783f33a010fcf24884c09504e782b38470869265b147/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/529c6003e1a5a7162162c41b33bbdfea681b750e27775b9077761351b7dc8d4c/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/713333c8d2447056c52570f44e105b038ae75c805ef501f4476b5636c24279b3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/154d3bd7aab4c392abaa09e41cc6ba716e7223dde4d897841d7ca37d0f92fa12/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c51968b1bda0188ac7201e39edebe795360bca3726042b738cfa6ac66e0f115a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9d5d639dd2c68561b36eec64b2526c8159c70e79ed6bb53dcbb9ca337996fc5c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/09743c5b48e4ed05778ac8dfac11b89adaefa594def05b0d2572767d021892dc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5af2a606266f7c7d190fe75788a1505d78a3599155417afa2fe115642d9abe70/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9928f109f0682fb7f14becbfab3d14e33e4b3c07b15c67ed6155043225ee8fc4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/d1e20389cdba697bb66c22f0589d5e53db652f7dad9dd01989088b95c56de88f/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/acd5866f18cb9a0b6f269cae624475b9086b973da0f17a22ee791f7a7945b325/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/141376d8ff487014e31c50baaddf99e7442fb0108fbea1712ae5f1c0737f4264/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/14b7e5f8f2c94200a3ccde10712c5b46f427a24e86a467aeb5ae53e7b6e4a59d/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/17539a02882e84860b79a1a1584b8860f3ed4437f0fa9d6660f23921cef9bf1b/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/7161db0f028a1bf590d752f92ff259078cf1c53ef19adecc99f192372b6e0878/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/16d129e35b03881a7b9f29ac4ff06cd68af9d8edf7d7ace21aa9ccd0f72290e9/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/2add8a75487848f0efebcaf6044d42f7fce365d8a3390f6c59ca8bf31692e0a7/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/15be28bd4b768460d43f1ae2aae69ad23188cae14447138e90293e9ce878d645/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/80d13e75282871767e8f151cf1157b1bbb403086c0fbf526ee8467aa6f69d3f5/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/d4b6d46702bce023d165531959e0bb7fd79a3e34b3741ae5a013ae199fcee4c3/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/306fc03ae3178ed0f0180bfc9e7b4a9a15b9950288d9477492ed978e1b24ddb4/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/821db4dc4c194d3f2d76987acdeaa5dc5df3534cbf48e027b16a7da214540e7a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ec7a626cd5c5225514385910f7595c8edf0e283d57e7002783cbff6e4dfceb55/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/13cc7bfbbdf211c2a33040a6431344de4d3164362127086c7d8f3e909b22354b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2d55b5ae0ca8722a844a81ae82e5d48daf3bb77574718b8c4e4e48dfa958e266/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a15bdfedcb8a81e1b34cb19a4c46b684d66411207f7b251a819f786c666cf21c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/31dbaf5c5658ca12f230d08dc1d47b2df525cadef1e3e8c7916fd75b5a4163f2/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/772e42d331b708364c6ecc6488df8c392b2bbe02bbd0ce1e7837255726a3e5e0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/9f7d611a087794ebf3898eee18ad1c6e210d820372ea2cd51b82a496d2324860/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/69f99ceead33d52558604c822a54e3b7be77f20b213e50b3a28ed5ac198b18c4/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.556997    4519 manager.go:217] Machine: {Timestamp:2026-10-30 19:34:35.556492048 +0000 UTC m=+0.057195763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105697 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec241a90ba3e10b05d0b0554a511c352 SystemUUID:ec241a90-ba3e-10b0-5d0b-0554a511c352 BootID:57ece507-41f2-4f0f-8b5a-703e85ac1e82 Filesystems:[{Device:/var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4bc0c6-9f96-4687-a645-5b727436c653/volumes/kubernetes.io~projected/kube-api-access-v762g DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7aac5a86-e546-4160-8867-5610087a1b42/volumes/kubernetes.io~projected/kube-api-access-cxrtk DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0bd988ff2ad535bad7a635ff0a44bc6f6614aa9ead821d09315a30a2f50a194/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28adba45-6c1c-4151-a898-94eaf4974f81/volumes/kubernetes.io~projected/kube-api-access-prfv4 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679/volumes/kubernetes.io~projected/kube-api-access-ffflj DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f5527b3ee17cecdc656ee10c5931b686b7553e1db105ed9627eab06c4a5a7e6/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volumes/kubernetes.io~projected/kube-api-access-lx8r5 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af9b796f616f8513c134b681bfeea6e16769b0ed5e37422fb8c0e8be8374b279/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0786b5ca278c4030659896f954bea884dab2ec315022f5392fdab26784c9614/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ea660b33bb399f60b597aa0552a5287568c5ea36bcee75969e142d6a5297dfa/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5/volumes/kubernetes.io~projected/kube-api-access-ltdch DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a037daa7-43b1-4565-8bf5-debb79042d0c/volumes/kubernetes.io~projected/kube-api-access-lc7tz DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d96c176-1899-4867-8855-09bfb02134e2/volumes/kubernetes.io~projected/kube-api-access-nwfql DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c620f89f08c9530f2397517431a8ee3ec99ce67a904cada33f82c575f2cddf2f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2ab193b546d6e627434bc6df3032bb57f82b33beeec27ad45063d9f1c9be0bb/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b290737d-48ee-4120-a1c8-46917d8f982c/volumes/kubernetes.io~projected/kube-api-access-hmrvv DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/072fe2ae46557f05ebf9b841780c4d07797e4fab742e614ad7a3a18ed7ebeec8/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b00be7b0-0f7e-4254-a128-e2fe6ee92277/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd498b84-3236-409b-8e94-e8ea178171d9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b00be7b0-0f7e-4254-a128-e2fe6ee92277/volumes/kubernetes.io~projected/kube-api-access-fgzk4 DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0403f76e-6189-4e37-bcb0-b8cecd653283/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1f3687bc80427c23e2fdfedf273d0e3d7a88794452e3eeede25087e729ed231/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd498b84-3236-409b-8e94-e8ea178171d9/volumes/kubernetes.io~projected/kube-api-access-hnlsb DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9967c4d42e0c33944033d31c6e2f04035710583e0d7b4c15dfb0c6eb7ac97cd8/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ae66028480815792c693695be196914e95bdeacc4913ce4470cfa5013875106/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9691d9-c7cd-4b98-8003-39cc09d90a02/volumes/kubernetes.io~projected/kube-api-access-2htz2 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f77ce3f3d68a73ad1c89fa3da4f05cf08e65255b69896c984c2eef9c68187d3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c37d572e-f147-442e-8b80-18a23ed12528/volumes/kubernetes.io~projected/kube-api-access-r9v9m DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51c72a9e-1e40-4f4d-ae0f-b42a4923c634/volumes/kubernetes.io~projected/kube-api-access-phw2p DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/734659d4-c70c-40be-8232-3f2d844704c1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1d813cb9fd093431a3df1c31caeebedcf3682b20cdf8569a470901f73c88afb/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35623b34-ad3c-4170-99a2-76ff61156c80/volumes/kubernetes.io~projected/kube-api-access-zhjrs DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e4e5602-243d-4855-9350-02d1c05762cf/volumes/kubernetes.io~projected/kube-api-access-mwl6c DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4f5527b3ee17cec MacAddress:9e:0b:6a:57:66:1f Speed:10000 Mtu:8901} {Name:9967c4d42e0c339 MacAddress:f6:82:61:c6:73:b3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d6:9c:59:37 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:56:3d:81:9c:a0 Speed:0 Mtu:8901} {Name:c1d813cb9fd0934 MacAddress:7a:80:a6:29:e0:fa Speed:10000 Mtu:8901} {Name:c8979816e3d3c4e MacAddress:ae:2b:38:e2:7d:e9 Speed:10000 Mtu:8901} {Name:d0bd988ff2ad535 MacAddress:4a:12:4d:c1:57:0c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d6:9c:59:37 Speed:0 Mtu:9001} {Name:f0786b5ca278c40 MacAddress:b6:55:bb:c6:03:4a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:e6:a1:b4:24:04:34 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:4c:2c:08:14:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557262    4519 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557304    4519 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557600    4519 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557619    4519 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557737    4519 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557745    4519 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557756    4519 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557770    4519 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.557851    4519 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.558043    4519 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.558148    4519 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.558185    4519 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.558194    4519 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.558203    4519 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.558213    4519 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.558221    4519 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.559181    4519 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.559195    4519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.559716    4519 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570139    4519 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570149    4519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570245    4519 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570430    4519 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570457    4519 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570564    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570577    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570584    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570590    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570596    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570602    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570608    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570614    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570622    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570627    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570642    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570651    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570677    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:34:35.573638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.570683    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:34:35.575787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.574000    4519 server.go:1263] "Started kubelet"
Oct 30 19:34:35.575787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.574162    4519 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:34:35.575787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.574152    4519 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:34:35.575787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.574520    4519 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:34:35.575787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.574840    4519 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:34:35.575787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.575647    4519 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:34:35.574555 ip-10-0-133-26 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.578202    4519 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.578290    4519 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.578305    4519 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:35.578319    4519 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.578200    4519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.578394    4519 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.578413    4519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:28:57 +0000 UTC" deadline="2028-05-04 03:23:58.404327902 +0000 UTC"
Oct 30 19:34:35.578768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.578526    4519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13231h49m22.825806647s"
Oct 30 19:34:35.580512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580458    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35623b34-ad3c-4170-99a2-76ff61156c80" volumeName="kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs" seLinuxMountContext=""
Oct 30 19:34:35.580613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580513    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" volumeName="kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:34:35.580613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580527    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="734659d4-c70c-40be-8232-3f2d844704c1" volumeName="kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:34:35.580613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580556    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="734659d4-c70c-40be-8232-3f2d844704c1" volumeName="kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-env-overrides" seLinuxMountContext=""
Oct 30 19:34:35.580613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580569    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aac5a86-e546-4160-8867-5610087a1b42" volumeName="kubernetes.io/configmap/7aac5a86-e546-4160-8867-5610087a1b42-config-volume" seLinuxMountContext=""
Oct 30 19:34:35.580613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580580    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aac5a86-e546-4160-8867-5610087a1b42" volumeName="kubernetes.io/empty-dir/7aac5a86-e546-4160-8867-5610087a1b42-tmp-dir" seLinuxMountContext=""
Oct 30 19:34:35.580613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580592    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b290737d-48ee-4120-a1c8-46917d8f982c" volumeName="kubernetes.io/configmap/b290737d-48ee-4120-a1c8-46917d8f982c-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:34:35.580613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580604    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35623b34-ad3c-4170-99a2-76ff61156c80" volumeName="kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580619    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d96c176-1899-4867-8855-09bfb02134e2" volumeName="kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580632    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a037daa7-43b1-4565-8bf5-debb79042d0c" volumeName="kubernetes.io/projected/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-api-access-lc7tz" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580644    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd498b84-3236-409b-8e94-e8ea178171d9" volumeName="kubernetes.io/projected/cd498b84-3236-409b-8e94-e8ea178171d9-kube-api-access-hnlsb" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580656    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0403f76e-6189-4e37-bcb0-b8cecd653283" volumeName="kubernetes.io/secret/0403f76e-6189-4e37-bcb0-b8cecd653283-agent-certs" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580668    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35623b34-ad3c-4170-99a2-76ff61156c80" volumeName="kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580683    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4bc0c6-9f96-4687-a645-5b727436c653" volumeName="kubernetes.io/projected/4e4bc0c6-9f96-4687-a645-5b727436c653-kube-api-access-v762g" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580706    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" volumeName="kubernetes.io/projected/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-kube-api-access-phw2p" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580719    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d96c176-1899-4867-8855-09bfb02134e2" volumeName="kubernetes.io/projected/9d96c176-1899-4867-8855-09bfb02134e2-kube-api-access-nwfql" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580733    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfd71eb-3976-4e6f-8bf2-182a1c9f22b5" volumeName="kubernetes.io/configmap/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-serviceca" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580746    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a47a9c2-1815-4bd6-b1bd-eb047f1cf679" volumeName="kubernetes.io/projected/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-kube-api-access-ffflj" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580758    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aac5a86-e546-4160-8867-5610087a1b42" volumeName="kubernetes.io/projected/7aac5a86-e546-4160-8867-5610087a1b42-kube-api-access-cxrtk" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580769    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e4e5602-243d-4855-9350-02d1c05762cf" volumeName="kubernetes.io/projected/8e4e5602-243d-4855-9350-02d1c05762cf-kube-api-access-mwl6c" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580781    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d96c176-1899-4867-8855-09bfb02134e2" volumeName="kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580794    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b00be7b0-0f7e-4254-a128-e2fe6ee92277" volumeName="kubernetes.io/secret/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-key" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580807    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c16bcfc9-20dd-4a12-ab7b-79dcbcb12557" volumeName="kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580819    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd498b84-3236-409b-8e94-e8ea178171d9" volumeName="kubernetes.io/secret/cd498b84-3236-409b-8e94-e8ea178171d9-cert" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580830    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0403f76e-6189-4e37-bcb0-b8cecd653283" volumeName="kubernetes.io/configmap/0403f76e-6189-4e37-bcb0-b8cecd653283-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580842    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9691d9-c7cd-4b98-8003-39cc09d90a02" volumeName="kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-tuned" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580854    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="734659d4-c70c-40be-8232-3f2d844704c1" volumeName="kubernetes.io/secret/734659d4-c70c-40be-8232-3f2d844704c1-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580865    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35623b34-ad3c-4170-99a2-76ff61156c80" volumeName="kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580890    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9691d9-c7cd-4b98-8003-39cc09d90a02" volumeName="kubernetes.io/projected/5f9691d9-c7cd-4b98-8003-39cc09d90a02-kube-api-access-2htz2" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580902    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="734659d4-c70c-40be-8232-3f2d844704c1" volumeName="kubernetes.io/projected/734659d4-c70c-40be-8232-3f2d844704c1-kube-api-access-lx8r5" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580914    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfd71eb-3976-4e6f-8bf2-182a1c9f22b5" volumeName="kubernetes.io/projected/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-kube-api-access-ltdch" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580926    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c37d572e-f147-442e-8b80-18a23ed12528" volumeName="kubernetes.io/projected/c37d572e-f147-442e-8b80-18a23ed12528-kube-api-access-r9v9m" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580951    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28adba45-6c1c-4151-a898-94eaf4974f81" volumeName="kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4" seLinuxMountContext=""
Oct 30 19:34:35.580945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580963    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b00be7b0-0f7e-4254-a128-e2fe6ee92277" volumeName="kubernetes.io/configmap/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-cabundle" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580975    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b00be7b0-0f7e-4254-a128-e2fe6ee92277" volumeName="kubernetes.io/projected/b00be7b0-0f7e-4254-a128-e2fe6ee92277-kube-api-access-fgzk4" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580987    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b290737d-48ee-4120-a1c8-46917d8f982c" volumeName="kubernetes.io/projected/b290737d-48ee-4120-a1c8-46917d8f982c-kube-api-access-hmrvv" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.580998    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" volumeName="kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.581009    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" volumeName="kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.581020    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a47a9c2-1815-4bd6-b1bd-eb047f1cf679" volumeName="kubernetes.io/empty-dir/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-tmp-dir" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.581032    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9691d9-c7cd-4b98-8003-39cc09d90a02" volumeName="kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-tmp" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.581044    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="734659d4-c70c-40be-8232-3f2d844704c1" volumeName="kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-config" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.581057    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a037daa7-43b1-4565-8bf5-debb79042d0c" volumeName="kubernetes.io/empty-dir/a037daa7-43b1-4565-8bf5-debb79042d0c-data-volume" seLinuxMountContext=""
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.581070    4519 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:34:35.582062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.581078    4519 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.583599    4519 factory.go:153] Registering CRI-O factory
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.583624    4519 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.583699    4519 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.583712    4519 factory.go:55] Registering systemd factory
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.583722    4519 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.583738    4519 factory.go:103] Registering Raw factory
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.583753    4519 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:34:35.584261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.584157    4519 manager.go:319] Starting recovery of all containers
Oct 30 19:34:35.602815 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.602746    4519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:34:35.604390 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.604364    4519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:34:35.604560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.604416    4519 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:34:35.604560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.604439    4519 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:34:35.604560 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:35.604516    4519 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:34:35.616998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.616962    4519 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="2fa816b819cb1234b67705e0443b59a2358297c5c2f910814ef5defd79b5dc0f" exitCode=0
Oct 30 19:34:35.616998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.616990    4519 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="669d74b6fc5b3fc014887348f243bbb36004342397fa8df83817e673a093bd6f" exitCode=0
Oct 30 19:34:35.616998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.616999    4519 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="e07783f8fd5db63dbc099f2fc0ce73a8a17b2a9e0dc2149d3bc4e8119e2ee1ee" exitCode=0
Oct 30 19:34:35.617226 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.617008    4519 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="9b58f8a81a3e28e0f64e882a5066d82f8c465090a3e91bbc49edde0f572f459a" exitCode=0
Oct 30 19:34:35.617226 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.617015    4519 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="a2195f83d58646051ea5dc073a0f976cc20b1f4ae2105e4e71df43a8e1ad5225" exitCode=0
Oct 30 19:34:35.617226 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.617023    4519 generic.go:356] "Generic (PLEG): container finished" podID="51c72a9e-1e40-4f4d-ae0f-b42a4923c634" containerID="aedb59cb97ab4dc9d8c44e2befce090534019553fd7bac52954ff1541159362e" exitCode=0
Oct 30 19:34:35.617725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.617702    4519 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:34:35.617725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.617718    4519 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:34:35.617856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.617718    4519 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:34:35.630047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.630024    4519 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:34:35.631520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.630595    4519 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:34:35.645098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.645068    4519 generic.go:356] "Generic (PLEG): container finished" podID="76e03f428e399a396d4b35504155fa47" containerID="e753c15a7134ecd9bfbf14a88c0f4fb50550451c8f09d0a72778b03de290e984" exitCode=0
Oct 30 19:34:35.656885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.656867    4519 manager.go:324] Recovery completed
Oct 30 19:34:35.674588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674531    4519 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:34:35.674588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674550    4519 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:34:35.674588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674571    4519 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:34:35.674760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674709    4519 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:34:35.674760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674724    4519 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:34:35.674760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674750    4519 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:34:35.674760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674759    4519 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:34:35.674868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674766    4519 policy_none.go:50] "Start"
Oct 30 19:34:35.674868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674783    4519 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:34:35.674868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.674795    4519 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:34:35.688094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.688063    4519 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:34:35.688221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.688102    4519 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:34:35.688221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.688118    4519 policy_none.go:44] "Start"
Oct 30 19:34:35.692778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.692305    4519 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:34:35.697699 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.697683    4519 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:34:35.697790 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:35.697713    4519 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:34:35.697790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.697725    4519 server.go:87] "Starting device plugin registration server"
Oct 30 19:34:35.698034 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.698020    4519 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:34:35.698093 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.698035    4519 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:34:35.698136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.698116    4519 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:34:35.698264 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.698223    4519 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:34:35.698320 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.698280    4519 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:34:35.698320 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.698288    4519 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:34:35.698419 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.698342    4519 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:34:35.698262983Z","UUID":"0749f95d-7192-478e-9357-14ed7345ade7","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:34:35.699536 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:35.699516    4519 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:34:35.699816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.699800    4519 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:34:35.699912 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.699826    4519 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:34:35.704641 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.704597    4519 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"]
Oct 30 19:34:35.705255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.705238    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.705349 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.705228    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" event={"ID":"d8c3ec59a3a8e5471fc2a192c7750dd6","Type":"ContainerStarted","Data":"80e316af4683e6dde93b73a032157a4c948d65ff9e5ec40524fadab1a3fbe5e6"}
Oct 30 19:34:35.705390 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.705356    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" event={"ID":"d8c3ec59a3a8e5471fc2a192c7750dd6","Type":"ContainerStarted","Data":"faf8e1643c66a6b45cf8d5a4641dfc35f25a1c808b7fb523d81511c09a6fa2d9"}
Oct 30 19:34:35.705437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.705399    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"76e03f428e399a396d4b35504155fa47","Type":"ContainerStarted","Data":"3078d5d78c6fcc23a3bce6885d49596d14d2ce75e82d2462cdd7072220a75028"}
Oct 30 19:34:35.705437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.705420    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"76e03f428e399a396d4b35504155fa47","Type":"ContainerDied","Data":"e753c15a7134ecd9bfbf14a88c0f4fb50550451c8f09d0a72778b03de290e984"}
Oct 30 19:34:35.705553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.705440    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"76e03f428e399a396d4b35504155fa47","Type":"ContainerStarted","Data":"47983612db6f93c711f1b73e05770c1755ada19c994a26fbd0ad14192a841b63"}
Oct 30 19:34:35.705553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.705449    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.728236 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.728216    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:34:35.728314 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:35.728256    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.729645 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.729633    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:34:35.729688 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:35.729658    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.798628 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.798602    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:34:35.799721 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.799704    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:34:35.799780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.799735    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:34:35.799780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.799745    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:34:35.799848 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.799780    4519 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.818856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.818833    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.818920 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.818863    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.818920 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.818880    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8c3ec59a3a8e5471fc2a192c7750dd6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"d8c3ec59a3a8e5471fc2a192c7750dd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.833507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.833475    4519 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.833581 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.833542    4519 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.919044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.919004    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8c3ec59a3a8e5471fc2a192c7750dd6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"d8c3ec59a3a8e5471fc2a192c7750dd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.919192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.919049    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.919192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.919074    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.919192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.919119    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.919192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.919124    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8c3ec59a3a8e5471fc2a192c7750dd6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"d8c3ec59a3a8e5471fc2a192c7750dd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:35.919192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:35.919124    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76e03f428e399a396d4b35504155fa47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"76e03f428e399a396d4b35504155fa47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:36.558550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.558523    4519 apiserver.go:52] "Watching apiserver"
Oct 30 19:34:36.598116 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.598089    4519 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:34:36.598599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.598577    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-htpxk","openshift-cluster-node-tuning-operator/tuned-rpc8d","openshift-multus/multus-xs9fh","openshift-multus/network-metrics-daemon-ktz22","openshift-network-operator/iptables-alerter-hsdk9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d","kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal","openshift-ingress-canary/ingress-canary-45g2c","openshift-dns/node-resolver-qmwpk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal","openshift-multus/multus-additional-cni-plugins-9mz4v","openshift-image-registry/node-ca-mb5hn","openshift-ovn-kubernetes/ovnkube-node-r5ssc","openshift-service-ca/service-ca-5fffc97bdf-5qphz","openshift-insights/periodic-gathering-h2cdk-f7vp9","openshift-network-diagnostics/network-check-target-292kr","openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv","kube-system/konnectivity-agent-zh9d6","openshift-dns/dns-default-5wrbn","openshift-insights/insights-runtime-extractor-7kstg"]
Oct 30 19:34:36.599801 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.599779    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:36.605154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.605130    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.606396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.606374    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.614445 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.614416    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.614445 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.614441    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.615018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.614999    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zv66j\""
Oct 30 19:34:36.615207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615185    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:34:36.615275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615221    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lhsbg\""
Oct 30 19:34:36.615275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615241    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:34:36.615275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615248    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.615275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615260    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t9cqt\""
Oct 30 19:34:36.615421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615405    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.615507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615456    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:34:36.615507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615456    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.615507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615486    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:34:36.615633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615508    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.615633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615536    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.615633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615543    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:34:36.615633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615600    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4c78\""
Oct 30 19:34:36.615633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615625    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.615763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615681    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.615763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615740    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52dmm\""
Oct 30 19:34:36.615817 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.615777    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fqfcc\""
Oct 30 19:34:36.623137 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.623123    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:34:36.623191 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.623143    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxzn8\""
Oct 30 19:34:36.623334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.623319    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qv89v\""
Oct 30 19:34:36.623393 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.623340    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xhvhw\""
Oct 30 19:34:36.623576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.623556    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:34:36.623643 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.623630    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:34:36.625175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625155    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:34:36.625175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625170    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:34:36.625275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625177    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.625275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625186    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.625275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625190    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:34:36.625275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625209    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.625275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625214    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:34:36.625529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625450    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.625529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625480    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.625529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625486    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.625529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625482    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:34:36.625529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625511    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.625833 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625601    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6fz68\""
Oct 30 19:34:36.625833 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625636    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.625945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625867    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:34:36.625945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625903    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.625945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625907    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7j4sn\""
Oct 30 19:34:36.625945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.625925    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.626168 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.626155    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk4qf\""
Oct 30 19:34:36.626218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.626169    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vj26l\""
Oct 30 19:34:36.626252 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.626213    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.626868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.626853    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:34:36.626916 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.626864    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:34:36.626916 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.626910    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:34:36.627194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627179    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hgf7z\""
Oct 30 19:34:36.627256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627182    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.627256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627216    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.627256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627242    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:34:36.627347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627275    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.627347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627283    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mtjn\""
Oct 30 19:34:36.627479 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627446    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:34:36.627576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627496    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.627576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627539    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hmkn8\""
Oct 30 19:34:36.627660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627622    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:34:36.627660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627638    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:34:36.627882 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627862    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-khwrx\""
Oct 30 19:34:36.627976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627901    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4mr2r\""
Oct 30 19:34:36.627976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627937    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:34:36.627976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.627974    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:34:36.637749 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.637732    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:34:36.647779 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.647754    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:36.647918 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.647885    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:36.668742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.668720    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:34:36.668866 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.668779    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:36.670322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.670304    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:34:36.670362 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.670347    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Oct 30 19:34:36.680537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.680516    4519 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:34:36.723391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723364    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdch\" (UniqueName: \"kubernetes.io/projected/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-kube-api-access-ltdch\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:34:36.723539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723395    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-etc-selinux\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.723539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723414    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aac5a86-e546-4160-8867-5610087a1b42-config-volume\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.723539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723429    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/7aac5a86-e546-4160-8867-5610087a1b42-kube-api-access-cxrtk\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.723539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723446    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-modprobe-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.723539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723461    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-var-lib-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.723539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723520    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.723757 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723564    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzk4\" (UniqueName: \"kubernetes.io/projected/b00be7b0-0f7e-4254-a128-e2fe6ee92277-kube-api-access-fgzk4\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:36.723794 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723764    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-device-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.723871 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723800    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-netns\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.723871 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723818    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-hosts-file\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:34:36.723871 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723856    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-tmp-dir\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:34:36.723960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723889    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffflj\" (UniqueName: \"kubernetes.io/projected/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-kube-api-access-ffflj\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:34:36.723960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723920    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-tmp-dir\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:34:36.723960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723938    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aac5a86-e546-4160-8867-5610087a1b42-config-volume\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.724089 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723955    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.724089 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.723986    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.724089 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724039    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-netd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.724089 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724072    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-env-overrides\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.724269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724111    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-conf\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.724269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724174    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-cni-binary-copy\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.724269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724252    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8r5\" (UniqueName: \"kubernetes.io/projected/734659d4-c70c-40be-8232-3f2d844704c1-kube-api-access-lx8r5\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.724360 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-original-pull-secret\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.724360 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724308    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b290737d-48ee-4120-a1c8-46917d8f982c-host-slash\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:34:36.724360 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724334    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.724360 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724354    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.724539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724404    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phw2p\" (UniqueName: \"kubernetes.io/projected/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-kube-api-access-phw2p\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.724598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724573    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd498b84-3236-409b-8e94-e8ea178171d9-cert\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:36.724664 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724592    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-cni-binary-copy\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.724724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724682    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.724724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724709    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-env-overrides\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.724822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724728    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.724822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724758    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a037daa7-43b1-4565-8bf5-debb79042d0c-crio-socket\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.724822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724759    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.724822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724782    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-systemd\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.725009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724831    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0403f76e-6189-4e37-bcb0-b8cecd653283-agent-certs\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:36.725009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724857    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/734659d4-c70c-40be-8232-3f2d844704c1-ovn-node-metrics-cert\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.725009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724892    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrvv\" (UniqueName: \"kubernetes.io/projected/b290737d-48ee-4120-a1c8-46917d8f982c-kube-api-access-hmrvv\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:34:36.725009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724924    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9v9m\" (UniqueName: \"kubernetes.io/projected/c37d572e-f147-442e-8b80-18a23ed12528-kube-api-access-r9v9m\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.725009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.724934    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd498b84-3236-409b-8e94-e8ea178171d9-cert\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:36.725009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725005    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/734659d4-c70c-40be-8232-3f2d844704c1-ovn-node-metrics-cert\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.725249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725039    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlsb\" (UniqueName: \"kubernetes.io/projected/cd498b84-3236-409b-8e94-e8ea178171d9-kube-api-access-hnlsb\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:36.725249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725070    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-kubelet-config\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.725249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725159    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-multus\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.725249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725185    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-conf-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.725249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725206    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-multus-daemon-config\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.725249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725208    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0403f76e-6189-4e37-bcb0-b8cecd653283-agent-certs\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:36.725409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725238    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-etc-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.725409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725385    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d96c176-1899-4867-8855-09bfb02134e2-multus-daemon-config\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.725918 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725902    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-serviceca\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:34:36.725949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725925    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.725949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725943    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v762g\" (UniqueName: \"kubernetes.io/projected/4e4bc0c6-9f96-4687-a645-5b727436c653-kube-api-access-v762g\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:36.726002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725958    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-lib-modules\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.726002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.725974    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.726072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726021    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aac5a86-e546-4160-8867-5610087a1b42-tmp-dir\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.726119 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726058    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-cnibin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.726171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726145    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aac5a86-e546-4160-8867-5610087a1b42-tmp-dir\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.726171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726162    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-bin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.726251 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726188    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.726251 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726194    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-serviceca\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:34:36.726251 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726232    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwl6c\" (UniqueName: \"kubernetes.io/projected/8e4e5602-243d-4855-9350-02d1c05762cf-kube-api-access-mwl6c\") pod \"migrator-66df5d496b-vr4bv\" (UID: \"8e4e5602-243d-4855-9350-02d1c05762cf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"
Oct 30 19:34:36.726366 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726297    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.726366 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726336    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.726438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726377    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-socket-dir-parent\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.726438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726407    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-hostroot\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.726520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726443    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-systemd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.726520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726483    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-config\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.726580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726520    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:36.726580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726551    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-multus-certs\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.726580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726568    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-systemd-units\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.726665 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726639    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-config\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.726665 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726645    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-log-socket\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.726744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726678    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-bin\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.726744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726709    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.726831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726753    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.726888 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726804    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-slash\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.726888 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726866    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-host\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.726977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726898    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysconfig\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.726977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726915    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.726977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726919    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.726977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726930    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-sys\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.726977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726948    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-k8s-cni-cncf-io\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.726977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726953    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.727201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.726978    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-script-lib\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.727201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727009    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.727201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727128    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-run\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.727201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727158    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-var-lib-kubelet\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.727201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727186    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-system-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.727337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727204    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/734659d4-c70c-40be-8232-3f2d844704c1-ovnkube-script-lib\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.727337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727213    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0403f76e-6189-4e37-bcb0-b8cecd653283-konnectivity-ca\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:36.727337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727290    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-ovn\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.727337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727317    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-node-log\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.727456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727334    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.727456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727364    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0403f76e-6189-4e37-bcb0-b8cecd653283-konnectivity-ca\") pod \"konnectivity-agent-zh9d6\" (UID: \"0403f76e-6189-4e37-bcb0-b8cecd653283\") " pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:34:36.727456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727378    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-dbus\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.727456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727407    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7tz\" (UniqueName: \"kubernetes.io/projected/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-api-access-lc7tz\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.727456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727430    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htz2\" (UniqueName: \"kubernetes.io/projected/5f9691d9-c7cd-4b98-8003-39cc09d90a02-kube-api-access-2htz2\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.727674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727500    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwfql\" (UniqueName: \"kubernetes.io/projected/9d96c176-1899-4867-8855-09bfb02134e2-kube-api-access-nwfql\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.727674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.727674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727602    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-host\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:34:36.727674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727634    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-key\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:36.727847 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727673    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:36.727847 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727748    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a037daa7-43b1-4565-8bf5-debb79042d0c-data-volume\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.727847 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727811    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-kubernetes\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.727966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727851    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-key\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:36.727966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727864    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.727966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a037daa7-43b1-4565-8bf5-debb79042d0c-data-volume\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.727966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b00be7b0-0f7e-4254-a128-e2fe6ee92277-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:36.727966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727922    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-kubelet\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.727966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727953    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.728160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.727982    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b290737d-48ee-4120-a1c8-46917d8f982c-iptables-alerter-script\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:34:36.728160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728002    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-socket-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.728160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728054    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-system-cni-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.728160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728075    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cnibin\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.728160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-tuned\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.728160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728154    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-tmp\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.728160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728159    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-tuned\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728191    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-kubelet\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728239    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9691d9-c7cd-4b98-8003-39cc09d90a02-tmp\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728239    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b290737d-48ee-4120-a1c8-46917d8f982c-iptables-alerter-script\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728248    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-registration-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728273    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-sys-fs\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728288    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728306    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-netns\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728341    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-etc-kubernetes\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.728433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.728384    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhjrs\" (UniqueName: \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.741947 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.741858    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffflj\" (UniqueName: \"kubernetes.io/projected/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-kube-api-access-ffflj\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:34:36.742075 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.741960    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdch\" (UniqueName: \"kubernetes.io/projected/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-kube-api-access-ltdch\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:34:36.742075 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.741970    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/7aac5a86-e546-4160-8867-5610087a1b42-kube-api-access-cxrtk\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.742537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.742514    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8r5\" (UniqueName: \"kubernetes.io/projected/734659d4-c70c-40be-8232-3f2d844704c1-kube-api-access-lx8r5\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.742624 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.742514    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzk4\" (UniqueName: \"kubernetes.io/projected/b00be7b0-0f7e-4254-a128-e2fe6ee92277-kube-api-access-fgzk4\") pod \"service-ca-5fffc97bdf-5qphz\" (UID: \"b00be7b0-0f7e-4254-a128-e2fe6ee92277\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz"
Oct 30 19:34:36.745046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.745027    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htz2\" (UniqueName: \"kubernetes.io/projected/5f9691d9-c7cd-4b98-8003-39cc09d90a02-kube-api-access-2htz2\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.745195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.745175    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhjrs\" (UniqueName: \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\") pod \"periodic-gathering-h2cdk-f7vp9\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") " pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:34:36.745630 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.745602    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrvv\" (UniqueName: \"kubernetes.io/projected/b290737d-48ee-4120-a1c8-46917d8f982c-kube-api-access-hmrvv\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:34:36.745828 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.745811    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v762g\" (UniqueName: \"kubernetes.io/projected/4e4bc0c6-9f96-4687-a645-5b727436c653-kube-api-access-v762g\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:36.745905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.745826    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prfv4\" (UniqueName: \"kubernetes.io/projected/28adba45-6c1c-4151-a898-94eaf4974f81-kube-api-access-prfv4\") pod \"network-check-target-292kr\" (UID: \"28adba45-6c1c-4151-a898-94eaf4974f81\") " pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:36.745905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.745871    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwfql\" (UniqueName: \"kubernetes.io/projected/9d96c176-1899-4867-8855-09bfb02134e2-kube-api-access-nwfql\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.745905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.745896    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwl6c\" (UniqueName: \"kubernetes.io/projected/8e4e5602-243d-4855-9350-02d1c05762cf-kube-api-access-mwl6c\") pod \"migrator-66df5d496b-vr4bv\" (UID: \"8e4e5602-243d-4855-9350-02d1c05762cf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vr4bv"
Oct 30 19:34:36.746546 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.746531    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phw2p\" (UniqueName: \"kubernetes.io/projected/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-kube-api-access-phw2p\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.746889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.746871    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlsb\" (UniqueName: \"kubernetes.io/projected/cd498b84-3236-409b-8e94-e8ea178171d9-kube-api-access-hnlsb\") pod \"ingress-canary-45g2c\" (UID: \"cd498b84-3236-409b-8e94-e8ea178171d9\") " pod="openshift-ingress-canary/ingress-canary-45g2c"
Oct 30 19:34:36.747069 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.747054    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9v9m\" (UniqueName: \"kubernetes.io/projected/c37d572e-f147-442e-8b80-18a23ed12528-kube-api-access-r9v9m\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.747227 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.747213    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7tz\" (UniqueName: \"kubernetes.io/projected/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-api-access-lc7tz\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.829086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829013    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-netns\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829047    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-hosts-file\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:34:36.829086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829064    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829080    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-netd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829103    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-conf\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829121    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-netns\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829125    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b290737d-48ee-4120-a1c8-46917d8f982c-host-slash\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829157    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b290737d-48ee-4120-a1c8-46917d8f982c-host-slash\") pod \"iptables-alerter-hsdk9\" (UID: \"b290737d-48ee-4120-a1c8-46917d8f982c\") " pod="openshift-network-operator/iptables-alerter-hsdk9"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829157    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a47a9c2-1815-4bd6-b1bd-eb047f1cf679-hosts-file\") pod \"node-resolver-qmwpk\" (UID: \"5a47a9c2-1815-4bd6-b1bd-eb047f1cf679\") " pod="openshift-dns/node-resolver-qmwpk"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829175    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829179    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829202    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a037daa7-43b1-4565-8bf5-debb79042d0c-crio-socket\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829224    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-systemd\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829197    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-netd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829239    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829249    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-kubelet-config\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829248    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a037daa7-43b1-4565-8bf5-debb79042d0c-crio-socket\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829291    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-systemd\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829293    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-kubelet-config\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-multus\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829279    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-multus\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829300    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-conf\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829327    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-conf-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829349    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-conf-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829352    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-etc-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829377    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829401    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-etc-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829403    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-lib-modules\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829448    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829486    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-cnibin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-bin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829532    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829546    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829555    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-socket-dir-parent\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829572    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-lib-modules\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829582    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-hostroot\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829588    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-cni-bin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829559    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-cnibin\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829605    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-systemd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829614    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-hostroot\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829626    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-multus-socket-dir-parent\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829635    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-multus-certs\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829658    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-systemd-units\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829678    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-systemd\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.829694    4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829679    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-log-socket\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829709    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-log-socket\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829702    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-systemd-units\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829725    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-bin\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829742    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-slash\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.829767    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.329751937 +0000 UTC m=+1.830455653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829678    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-multus-certs\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829781    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-slash\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829782    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-cni-bin\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-host\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829831    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysconfig\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829849    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829872    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-sys\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829883    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysconfig\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829885    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-host\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829894    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-k8s-cni-cncf-io\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829930    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.829932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829953    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-k8s-cni-cncf-io\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829958    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-run\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829975    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-sysctl-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-var-lib-kubelet\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.829933    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-sys\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830008    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-system-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830030    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-ovn\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830012    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-run\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830044    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-node-log\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830046    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-system-cni-dir\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830068    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-run-ovn\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830013    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-var-lib-kubelet\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830081    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830086    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-node-log\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830104    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-dbus\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830134    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-host\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830160    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-kubernetes\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830167    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830183    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-kubelet\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830187    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfd71eb-3976-4e6f-8bf2-182a1c9f22b5-host\") pod \"node-ca-mb5hn\" (UID: \"adfd71eb-3976-4e6f-8bf2-182a1c9f22b5\") " pod="openshift-image-registry/node-ca-mb5hn"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830203    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16bcfc9-20dd-4a12-ab7b-79dcbcb12557-dbus\") pod \"global-pull-secret-syncer-htpxk\" (UID: \"c16bcfc9-20dd-4a12-ab7b-79dcbcb12557\") " pod="kube-system/global-pull-secret-syncer-htpxk"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830209    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830230    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-var-lib-kubelet\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830209    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-kubernetes\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830239    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-socket-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830254    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830275    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-system-cni-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830290    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-socket-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830293    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cnibin\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830311    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-cnibin\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830331    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-kubelet\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830341    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51c72a9e-1e40-4f4d-ae0f-b42a4923c634-system-cni-dir\") pod \"multus-additional-cni-plugins-9mz4v\" (UID: \"51c72a9e-1e40-4f4d-ae0f-b42a4923c634\") " pod="openshift-multus/multus-additional-cni-plugins-9mz4v"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830355    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-registration-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830363    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-host-kubelet\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830381    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-sys-fs\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830402    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-registration-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830407    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830438    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-netns\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830452    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-sys-fs\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830490    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a037daa7-43b1-4565-8bf5-debb79042d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830496    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-etc-kubernetes\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830504    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-host-run-netns\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.830514    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830529    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-etc-selinux\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830542    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d96c176-1899-4867-8855-09bfb02134e2-etc-kubernetes\") pod \"multus-xs9fh\" (UID: \"9d96c176-1899-4867-8855-09bfb02134e2\") " pod="openshift-multus/multus-xs9fh"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.830599    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.330585546 +0000 UTC m=+1.831289264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : secret "metrics-daemon-secret" not found
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830609    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-etc-selinux\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830611    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-modprobe-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830640    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-var-lib-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830662    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830668    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9691d9-c7cd-4b98-8003-39cc09d90a02-etc-modprobe-d\") pod \"tuned-rpc8d\" (UID: \"5f9691d9-c7cd-4b98-8003-39cc09d90a02\") " pod="openshift-cluster-node-tuning-operator/tuned-rpc8d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830687    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-device-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830688    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/734659d4-c70c-40be-8232-3f2d844704c1-var-lib-openvswitch\") pod \"ovnkube-node-r5ssc\" (UID: \"734659d4-c70c-40be-8232-3f2d844704c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.830747    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c37d572e-f147-442e-8b80-18a23ed12528-device-dir\") pod \"aws-ebs-csi-driver-node-5wb7d\" (UID: \"c37d572e-f147-442e-8b80-18a23ed12528\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5wb7d"
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.830791    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:36.831218 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:36.830859    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.330843032 +0000 UTC m=+1.831546754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:36.908795 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:36.908780    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:34:37.184092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:37.184066    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:37.262092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:37.262060    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:37.334646 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:37.334611    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:37.334806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:37.334663    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:37.334806 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:37.334750    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:34:37.334806 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:37.334752    4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:37.334806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:37.334784    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:37.334806 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:37.334793    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.334781167 +0000 UTC m=+2.835484886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : secret "metrics-daemon-secret" not found
Oct 30 19:34:37.335071 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:37.334816    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.334804408 +0000 UTC m=+2.835508125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:37.335071 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:37.334861    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:37.335071 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:37.334907    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.334897597 +0000 UTC m=+2.835601299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:37.658430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:37.658404    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qmwpk_5a47a9c2-1815-4bd6-b1bd-eb047f1cf679/dns-node-resolver/0.log"
Oct 30 19:34:38.342766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:38.342724    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:38.342958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:38.342784    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:38.342958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:38.342809    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:38.342958 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:38.342878    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:34:38.342958 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:38.342896    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:38.342958 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:38.342941    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.34292792 +0000 UTC m=+4.843631622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:38.342958 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:38.342952    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.342946443 +0000 UTC m=+4.843650145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : secret "metrics-daemon-secret" not found
Oct 30 19:34:38.342958 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:38.342878    4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:38.343169 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:38.342974    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls podName:a037daa7-43b1-4565-8bf5-debb79042d0c nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.342969209 +0000 UTC m=+4.843672911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kstg" (UID: "a037daa7-43b1-4565-8bf5-debb79042d0c") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:34:38.652791 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:38.652710    4519 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:34:38.658661 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:38.658626    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mb5hn_adfd71eb-3976-4e6f-8bf2-182a1c9f22b5/node-ca/0.log"
Oct 30 19:34:39.058897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:39.058872    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-45g2c_cd498b84-3236-409b-8e94-e8ea178171d9/serve-healthcheck-canary/0.log"
Oct 30 19:34:39.656245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:39.656166    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5qphz" event={"ID":"b00be7b0-0f7e-4254-a128-e2fe6ee92277","Type":"ContainerStarted","Data":"a82d3c6f32114fc1f6b46d607e91b53ad18c404107a7b1d16a86618146ec17d4"}
Oct 30 19:34:39.659385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:39.659357    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vr4bv_8e4e5602-243d-4855-9350-02d1c05762cf/migrator/0.log"
Oct 30 19:34:40.361212 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.361176    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:40.361396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.361229    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:40.361396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.361253    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:40.361396 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:40.361336    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:34:40.361396 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:40.361341    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:34:40.361396 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:40.361387    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls podName:7aac5a86-e546-4160-8867-5610087a1b42 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.361375298 +0000 UTC m=+8.862079005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls") pod "dns-default-5wrbn" (UID: "7aac5a86-e546-4160-8867-5610087a1b42") : secret "dns-default-metrics-tls" not found
Oct 30 19:34:40.361396 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:40.361400    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.361394213 +0000 UTC m=+8.862097918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : secret "metrics-daemon-secret" not found
Oct 30 19:34:40.361640 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.361606    4519 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:34:40.363668 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.363648    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a037daa7-43b1-4565-8bf5-debb79042d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kstg\" (UID: \"a037daa7-43b1-4565-8bf5-debb79042d0c\") " pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:40.507597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.507563    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kstg"
Oct 30 19:34:40.741568 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.741535    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:34:40.801913 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:40.801883    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kstg"]
Oct 30 19:34:40.803057 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:40.803032    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda037daa7_43b1_4565_8bf5_debb79042d0c.slice/crio-d2ff66af649dae49b39b10d8805e7d16f4c4635f0294a7a67ab4eff833362213 WatchSource:0}: Error finding container d2ff66af649dae49b39b10d8805e7d16f4c4635f0294a7a67ab4eff833362213: Status 404 returned error can't find the container with id d2ff66af649dae49b39b10d8805e7d16f4c4635f0294a7a67ab4eff833362213
Oct 30 19:34:41.663163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:41.663133    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kstg" event={"ID":"a037daa7-43b1-4565-8bf5-debb79042d0c","Type":"ContainerStarted","Data":"6edfb377aaa0e870c5dd6fbcee544fdb9348bfe3e977693fd96e73093806b59a"}
Oct 30 19:34:41.663276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:41.663186    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kstg" event={"ID":"a037daa7-43b1-4565-8bf5-debb79042d0c","Type":"ContainerStarted","Data":"3c348ec91b6459f2aa3389c6322ecb8cdeaf786fe483156ed57335d954f864fc"}
Oct 30 19:34:41.663276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:41.663207    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kstg" event={"ID":"a037daa7-43b1-4565-8bf5-debb79042d0c","Type":"ContainerStarted","Data":"d2ff66af649dae49b39b10d8805e7d16f4c4635f0294a7a67ab4eff833362213"}
Oct 30 19:34:43.670848 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:43.670811    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kstg" event={"ID":"a037daa7-43b1-4565-8bf5-debb79042d0c","Type":"ContainerStarted","Data":"56cb2c39f187d2cf1bf72ec4601af3b95cd71b187b6f4edf5058a5b7d3d5e3ba"}
Oct 30 19:34:44.389109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:44.389073    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:44.389109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:44.389112    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:44.389328 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:44.389220    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:34:44.389328 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:34:44.389286    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs podName:4e4bc0c6-9f96-4687-a645-5b727436c653 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:52.389272352 +0000 UTC m=+16.889976072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs") pod "network-metrics-daemon-ktz22" (UID: "4e4bc0c6-9f96-4687-a645-5b727436c653") : secret "metrics-daemon-secret" not found
Oct 30 19:34:44.391339 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:44.391320    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aac5a86-e546-4160-8867-5610087a1b42-metrics-tls\") pod \"dns-default-5wrbn\" (UID: \"7aac5a86-e546-4160-8867-5610087a1b42\") " pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:44.406476 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:44.406440    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:44.698254 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:44.698225    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5wrbn"]
Oct 30 19:34:44.699558 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:44.699530    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aac5a86_e546_4160_8867_5610087a1b42.slice/crio-e6540ccfa242ed2db2a18edc424f91be0242daab9bef1e01094294a55cbed963 WatchSource:0}: Error finding container e6540ccfa242ed2db2a18edc424f91be0242daab9bef1e01094294a55cbed963: Status 404 returned error can't find the container with id e6540ccfa242ed2db2a18edc424f91be0242daab9bef1e01094294a55cbed963
Oct 30 19:34:45.678358 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:45.678321    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wrbn" event={"ID":"7aac5a86-e546-4160-8867-5610087a1b42","Type":"ContainerStarted","Data":"e6540ccfa242ed2db2a18edc424f91be0242daab9bef1e01094294a55cbed963"}
Oct 30 19:34:46.682743 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:46.682706    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wrbn" event={"ID":"7aac5a86-e546-4160-8867-5610087a1b42","Type":"ContainerStarted","Data":"5562e2b730465bc30ec5658b293e9dcea6d9996d3e99c921653948e61e2842d9"}
Oct 30 19:34:46.683133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:46.682799    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wrbn" event={"ID":"7aac5a86-e546-4160-8867-5610087a1b42","Type":"ContainerStarted","Data":"c25603887cf2e682bfa480a6f014f8eb40c73fd1a1e5fd0886668afc2565168b"}
Oct 30 19:34:46.683133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:46.682831    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:50.622897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:50.622863    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r5ssc"
Oct 30 19:34:52.447187 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:52.447144    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:52.449679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:52.449650    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4bc0c6-9f96-4687-a645-5b727436c653-metrics-certs\") pod \"network-metrics-daemon-ktz22\" (UID: \"4e4bc0c6-9f96-4687-a645-5b727436c653\") " pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:52.501106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:52.501071    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktz22"
Oct 30 19:34:52.792787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:52.792757    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ktz22"]
Oct 30 19:34:52.794247 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:52.794219    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4bc0c6_9f96_4687_a645_5b727436c653.slice/crio-775c31645eb3528102436b3b930ae139871e71189de830a7a000187d00d10cad WatchSource:0}: Error finding container 775c31645eb3528102436b3b930ae139871e71189de830a7a000187d00d10cad: Status 404 returned error can't find the container with id 775c31645eb3528102436b3b930ae139871e71189de830a7a000187d00d10cad
Oct 30 19:34:53.706319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:53.706285    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ktz22" event={"ID":"4e4bc0c6-9f96-4687-a645-5b727436c653","Type":"ContainerStarted","Data":"775c31645eb3528102436b3b930ae139871e71189de830a7a000187d00d10cad"}
Oct 30 19:34:54.051482 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.051393    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-kqqc8"]
Oct 30 19:34:54.054609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.054588    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.074288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.074267    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:34:54.074288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.074281    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:34:54.085969 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.085948    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-rgtls\""
Oct 30 19:34:54.086612 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.086591    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kqqc8"]
Oct 30 19:34:54.151422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.151391    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b665c84d-sdhgp"]
Oct 30 19:34:54.154405 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.154387    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.159931 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.159904    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5-tmp\") pod \"downloads-6dd789c8b7-kqqc8\" (UID: \"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5\") " pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.160033 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.159932    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7m6\" (UniqueName: \"kubernetes.io/projected/f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5-kube-api-access-jg7m6\") pod \"downloads-6dd789c8b7-kqqc8\" (UID: \"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5\") " pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.183822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.183796    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:34:54.183941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.183830    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:34:54.183941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.183875    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:34:54.184018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.183941    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:34:54.195314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.195288    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2hnnk\""
Oct 30 19:34:54.196001 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.195984    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b665c84d-sdhgp"]
Oct 30 19:34:54.260671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260644    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-registry-tls\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.260843 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260677    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-trusted-ca\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.260843 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260700    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstzz\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-kube-api-access-cstzz\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.260843 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260778    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-installation-pull-secrets\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.260955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260875    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5-tmp\") pod \"downloads-6dd789c8b7-kqqc8\" (UID: \"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5\") " pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.260955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260897    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7m6\" (UniqueName: \"kubernetes.io/projected/f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5-kube-api-access-jg7m6\") pod \"downloads-6dd789c8b7-kqqc8\" (UID: \"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5\") " pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.260955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260923    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-registry-certificates\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.260955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.260945    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-ca-trust-extracted\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.261089 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.261007    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-bound-sa-token\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.261089 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.261070    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-image-registry-private-configuration\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.261347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.261329    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5-tmp\") pod \"downloads-6dd789c8b7-kqqc8\" (UID: \"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5\") " pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.299037 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.299014    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7m6\" (UniqueName: \"kubernetes.io/projected/f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5-kube-api-access-jg7m6\") pod \"downloads-6dd789c8b7-kqqc8\" (UID: \"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5\") " pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.362235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362173    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-ca-trust-extracted\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362201    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-bound-sa-token\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362222    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-image-registry-private-configuration\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362252    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-registry-tls\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362281    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-trusted-ca\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362314    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstzz\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-kube-api-access-cstzz\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362346    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-installation-pull-secrets\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362382    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-registry-certificates\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.362763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.362638    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-ca-trust-extracted\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.363561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.363537    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-trusted-ca\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.363672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.363601    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-registry-certificates\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.364857 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.364831    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-installation-pull-secrets\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.364857 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.364851    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-image-registry-private-configuration\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.364980 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.364891    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-registry-tls\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.365675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.365663    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:34:54.381268 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.381237    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstzz\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-kube-api-access-cstzz\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.381658 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.381638    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c-bound-sa-token\") pod \"image-registry-7b665c84d-sdhgp\" (UID: \"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c\") " pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.465147 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.465116    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:54.681334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.681304    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kqqc8"]
Oct 30 19:34:54.682714 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:54.682681    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28b4d2c_a0a6_4ca0_bec4_f17c563b0de5.slice/crio-1a8e7931cd5c8bb22ebda88a0246869d8387115b3dbfa3b6820df15e4fd393a6 WatchSource:0}: Error finding container 1a8e7931cd5c8bb22ebda88a0246869d8387115b3dbfa3b6820df15e4fd393a6: Status 404 returned error can't find the container with id 1a8e7931cd5c8bb22ebda88a0246869d8387115b3dbfa3b6820df15e4fd393a6
Oct 30 19:34:54.710288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.710258    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kqqc8" event={"ID":"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5","Type":"ContainerStarted","Data":"1a8e7931cd5c8bb22ebda88a0246869d8387115b3dbfa3b6820df15e4fd393a6"}
Oct 30 19:34:54.712069 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.712046    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ktz22" event={"ID":"4e4bc0c6-9f96-4687-a645-5b727436c653","Type":"ContainerStarted","Data":"2a060cc7ee5a33d45e792031b96a89408b8f1e1eb509ea2f747d6324c7355694"}
Oct 30 19:34:54.712151 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.712088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ktz22" event={"ID":"4e4bc0c6-9f96-4687-a645-5b727436c653","Type":"ContainerStarted","Data":"83b3f12858ea0a70aff7dc183f8f95a84ba2bff1f409c2eaec5e24bf7159efb6"}
Oct 30 19:34:54.761088 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:54.761064    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b665c84d-sdhgp"]
Oct 30 19:34:54.763098 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:54.763075    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2c8fff_0ca4_45b8_a9a8_1eaeadcf2c2c.slice/crio-7cd99cb41aaf34fd58c865d7465916dc10e4f2e7c6b17aa3d44e14ea9a9c3fff WatchSource:0}: Error finding container 7cd99cb41aaf34fd58c865d7465916dc10e4f2e7c6b17aa3d44e14ea9a9c3fff: Status 404 returned error can't find the container with id 7cd99cb41aaf34fd58c865d7465916dc10e4f2e7c6b17aa3d44e14ea9a9c3fff
Oct 30 19:34:55.716547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:55.716500    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b665c84d-sdhgp" event={"ID":"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c","Type":"ContainerStarted","Data":"b185f5ead49a03c4ef58510c64095077e184e29f85b1c4e1ce17eb52b1a79225"}
Oct 30 19:34:55.716995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:55.716567    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b665c84d-sdhgp" event={"ID":"bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c","Type":"ContainerStarted","Data":"7cd99cb41aaf34fd58c865d7465916dc10e4f2e7c6b17aa3d44e14ea9a9c3fff"}
Oct 30 19:34:56.231079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.231016    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b665c84d-sdhgp" podStartSLOduration=2.231000513 podStartE2EDuration="2.231000513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:55.766863416 +0000 UTC m=+20.267567141" watchObservedRunningTime="2026-10-30 19:34:56.231000513 +0000 UTC m=+20.731704234"
Oct 30 19:34:56.231581 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.231557    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"]
Oct 30 19:34:56.234865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.234845    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"
Oct 30 19:34:56.252737 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.252712    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5pqxf\""
Oct 30 19:34:56.255202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.255185    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 19:34:56.256668 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.256560    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"]
Oct 30 19:34:56.373532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.373495    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bdf5a7ee-c3fa-4650-b424-9fdb15c7e742-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2v84p\" (UID: \"bdf5a7ee-c3fa-4650-b424-9fdb15c7e742\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"
Oct 30 19:34:56.474735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.474698    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bdf5a7ee-c3fa-4650-b424-9fdb15c7e742-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2v84p\" (UID: \"bdf5a7ee-c3fa-4650-b424-9fdb15c7e742\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"
Oct 30 19:34:56.477505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.477461    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bdf5a7ee-c3fa-4650-b424-9fdb15c7e742-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2v84p\" (UID: \"bdf5a7ee-c3fa-4650-b424-9fdb15c7e742\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"
Oct 30 19:34:56.547677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.547588    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"
Oct 30 19:34:56.688889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.688854    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5wrbn"
Oct 30 19:34:56.720493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.720450    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:34:56.850645 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:56.850613    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"]
Oct 30 19:34:56.852233 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:34:56.852208    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf5a7ee_c3fa_4650_b424_9fdb15c7e742.slice/crio-6efc0deabd67d5ec829e450bcec399c477645bb863f4cec45a43616caa2366b0 WatchSource:0}: Error finding container 6efc0deabd67d5ec829e450bcec399c477645bb863f4cec45a43616caa2366b0: Status 404 returned error can't find the container with id 6efc0deabd67d5ec829e450bcec399c477645bb863f4cec45a43616caa2366b0
Oct 30 19:34:57.726478 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:57.726431    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p" event={"ID":"bdf5a7ee-c3fa-4650-b424-9fdb15c7e742","Type":"ContainerStarted","Data":"6efc0deabd67d5ec829e450bcec399c477645bb863f4cec45a43616caa2366b0"}
Oct 30 19:34:58.731734 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:58.731694    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p" event={"ID":"bdf5a7ee-c3fa-4650-b424-9fdb15c7e742","Type":"ContainerStarted","Data":"6c78be5717c5d50acd5d46d2a2c78ecc640fc0302ecb0f5018eed03b69f5e3ea"}
Oct 30 19:34:58.732133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:58.731906    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"
Oct 30 19:34:58.737041 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:58.737015    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p"
Oct 30 19:34:58.773489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:58.772908    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2v84p" podStartSLOduration=1.778476521 podStartE2EDuration="2.772892914s" totalImagesPullingTime="994.416393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:58.772365997 +0000 UTC m=+23.273069722" watchObservedRunningTime="2026-10-30 19:34:58.772892914 +0000 UTC m=+23.273596637"
Oct 30 19:34:59.490891 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.490856    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"]
Oct 30 19:34:59.494454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.494431    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.521760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.521725    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:34:59.521901 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.521821    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:34:59.521901 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.521893    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:34:59.521965 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.521944    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2sbx9\""
Oct 30 19:34:59.522012 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.521978    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:34:59.522064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.522016    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:34:59.534776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.534594    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"]
Oct 30 19:34:59.601778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.601743    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ed79173-91c5-49da-9e0c-4818d7ad75ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.601988 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.601800    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ed79173-91c5-49da-9e0c-4818d7ad75ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.601988 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.601886    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkrl\" (UniqueName: \"kubernetes.io/projected/6ed79173-91c5-49da-9e0c-4818d7ad75ee-kube-api-access-spkrl\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.601988 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.601950    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ed79173-91c5-49da-9e0c-4818d7ad75ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.702528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.702495    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ed79173-91c5-49da-9e0c-4818d7ad75ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.702528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.702532    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkrl\" (UniqueName: \"kubernetes.io/projected/6ed79173-91c5-49da-9e0c-4818d7ad75ee-kube-api-access-spkrl\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.702785 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.702572    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ed79173-91c5-49da-9e0c-4818d7ad75ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.702785 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.702638    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ed79173-91c5-49da-9e0c-4818d7ad75ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.703418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.703394    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ed79173-91c5-49da-9e0c-4818d7ad75ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.705316 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.705281    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ed79173-91c5-49da-9e0c-4818d7ad75ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.705859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.705830    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ed79173-91c5-49da-9e0c-4818d7ad75ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.722341 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.722319    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkrl\" (UniqueName: \"kubernetes.io/projected/6ed79173-91c5-49da-9e0c-4818d7ad75ee-kube-api-access-spkrl\") pod \"prometheus-operator-9cd89b74c-xfn9z\" (UID: \"6ed79173-91c5-49da-9e0c-4818d7ad75ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:34:59.811065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:34:59.810985    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"
Oct 30 19:35:00.112439 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:00.112409    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z"]
Oct 30 19:35:00.114870 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:35:00.114821    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed79173_91c5_49da_9e0c_4818d7ad75ee.slice/crio-aea7bb585c684d39b4844d94f67ab7cbc59a9d0ccb0ed2aad7099e42f717d3a5 WatchSource:0}: Error finding container aea7bb585c684d39b4844d94f67ab7cbc59a9d0ccb0ed2aad7099e42f717d3a5: Status 404 returned error can't find the container with id aea7bb585c684d39b4844d94f67ab7cbc59a9d0ccb0ed2aad7099e42f717d3a5
Oct 30 19:35:00.741169 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:00.741090    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z" event={"ID":"6ed79173-91c5-49da-9e0c-4818d7ad75ee","Type":"ContainerStarted","Data":"aea7bb585c684d39b4844d94f67ab7cbc59a9d0ccb0ed2aad7099e42f717d3a5"}
Oct 30 19:35:01.746602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:01.746523    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z" event={"ID":"6ed79173-91c5-49da-9e0c-4818d7ad75ee","Type":"ContainerStarted","Data":"95ebdf0756b70c87c6960c54eca1f0adea1bad6fefb718c819571d715f5634a8"}
Oct 30 19:35:01.746602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:01.746588    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z" event={"ID":"6ed79173-91c5-49da-9e0c-4818d7ad75ee","Type":"ContainerStarted","Data":"13de2c5ab6c56ca4516a1591d3fa0f5e5da2636eff8c592cf0b8a38d8ad8ffd2"}
Oct 30 19:35:01.787214 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:01.787166    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xfn9z" podStartSLOduration=1.4522523569999999 podStartE2EDuration="2.787152443s" totalImagesPullingTime="1.334900086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:01.786834746 +0000 UTC m=+26.287538471" watchObservedRunningTime="2026-10-30 19:35:01.787152443 +0000 UTC m=+26.287856167"
Oct 30 19:35:02.231163 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:35:02.231129    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:35:04.207165 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.207128    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mrvbg"]
Oct 30 19:35:04.218213 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.218178    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.243167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.243145    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:35:04.266675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.266650    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:35:04.266675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.266650    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:35:04.266851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.266742    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dpldx\""
Oct 30 19:35:04.339030 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.338997    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-sys\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-textfile\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339072    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339095    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-root\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339125    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-tls\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339144    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpbq\" (UniqueName: \"kubernetes.io/projected/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-kube-api-access-6mpbq\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339167    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-wtmp\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339182    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.339485 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.339208    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-metrics-client-ca\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.439933 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.439898    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.439939    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-root\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440007    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-tls\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440016    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-root\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440141    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpbq\" (UniqueName: \"kubernetes.io/projected/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-kube-api-access-6mpbq\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440194    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-wtmp\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440225    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440265    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-metrics-client-ca\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440313    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-sys\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440351    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-textfile\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440363    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-wtmp\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440426    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-sys\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.440722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440698    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-textfile\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.441018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.440995    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-metrics-client-ca\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.441090 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.441051    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.442605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.442585    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-tls\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.442699 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.442646    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.463423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.463360    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpbq\" (UniqueName: \"kubernetes.io/projected/951e08f8-4211-4fd9-b759-1ebd5a2ccdc6-kube-api-access-6mpbq\") pod \"node-exporter-mrvbg\" (UID: \"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6\") " pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.536266 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.536228    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mrvbg"
Oct 30 19:35:04.550451 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:35:04.550426    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951e08f8_4211_4fd9_b759_1ebd5a2ccdc6.slice/crio-0f45cdc60a4e78c131e94e7aaedae4da72a54e380c694d43ff9de1a298b74556 WatchSource:0}: Error finding container 0f45cdc60a4e78c131e94e7aaedae4da72a54e380c694d43ff9de1a298b74556: Status 404 returned error can't find the container with id 0f45cdc60a4e78c131e94e7aaedae4da72a54e380c694d43ff9de1a298b74556
Oct 30 19:35:04.759547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:04.759451    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrvbg" event={"ID":"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6","Type":"ContainerStarted","Data":"0f45cdc60a4e78c131e94e7aaedae4da72a54e380c694d43ff9de1a298b74556"}
Oct 30 19:35:05.416813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:05.416786    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zh9d6"
Oct 30 19:35:05.765426 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:05.765342    4519 generic.go:356] "Generic (PLEG): container finished" podID="951e08f8-4211-4fd9-b759-1ebd5a2ccdc6" containerID="a436c2939d92c1245cf886be170255479f0f9c0c2899472b1368ae275a7dfa2b" exitCode=0
Oct 30 19:35:05.765426 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:05.765416    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrvbg" event={"ID":"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6","Type":"ContainerDied","Data":"a436c2939d92c1245cf886be170255479f0f9c0c2899472b1368ae275a7dfa2b"}
Oct 30 19:35:06.772146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:06.772103    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrvbg" event={"ID":"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6","Type":"ContainerStarted","Data":"91a7e546a7f1edc505e488644e10c80940e64cc051904a48bcafb127df5a11d2"}
Oct 30 19:35:06.772716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:06.772208    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mrvbg" event={"ID":"951e08f8-4211-4fd9-b759-1ebd5a2ccdc6","Type":"ContainerStarted","Data":"806cc59b931042133ca90187e6db78b49c96f8b48b7bfd1420de08870bab312e"}
Oct 30 19:35:06.816554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:06.816500    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mrvbg" podStartSLOduration=1.84558874 podStartE2EDuration="2.816482437s" totalImagesPullingTime="890.92612ms" totalInitContainerRuntime="79.967577ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:06.816432577 +0000 UTC m=+31.317136300" watchObservedRunningTime="2026-10-30 19:35:06.816482437 +0000 UTC m=+31.317186155"
Oct 30 19:35:10.744721 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:10.744689    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-292kr"
Oct 30 19:35:11.794457 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:11.794413    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kqqc8" event={"ID":"f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5","Type":"ContainerStarted","Data":"598730571d01502804828bed9be99b7ee227578dba61532a19d6b87533c025e1"}
Oct 30 19:35:11.794968 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:11.794729    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:35:11.805166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:11.805135    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kqqc8"
Oct 30 19:35:11.857940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:11.857880    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-kqqc8" podStartSLOduration=1.633296762 podStartE2EDuration="17.857862014s" totalImagesPullingTime="16.224565252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:11.845400407 +0000 UTC m=+36.346104134" watchObservedRunningTime="2026-10-30 19:35:11.857862014 +0000 UTC m=+36.358565741"
Oct 30 19:35:12.388449 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.388414    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78bc879ccd-wpwzs"]
Oct 30 19:35:12.422865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.422828    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bc879ccd-wpwzs"]
Oct 30 19:35:12.423047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.422973    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.437987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.437962    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 19:35:12.449997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.449871    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 19:35:12.449997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.449883    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g55w6\""
Oct 30 19:35:12.449997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.449868    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 19:35:12.449997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.449915    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 19:35:12.450508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.450485    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 19:35:12.474557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.474527    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 19:35:12.507702 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.507667    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-oauth-config\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.507863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.507733    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-serving-cert\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.507863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.507793    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f552d66-611f-4118-a361-74964e660f36-tmp\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.507863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.507830    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t62nz\" (UniqueName: \"kubernetes.io/projected/4f552d66-611f-4118-a361-74964e660f36-kube-api-access-t62nz\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.508016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.507867    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-service-ca\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.508016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.507891    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-oauth-serving-cert\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.508121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.508012    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-trusted-ca-bundle\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.508121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.508052    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-console-config\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609358 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609316    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62nz\" (UniqueName: \"kubernetes.io/projected/4f552d66-611f-4118-a361-74964e660f36-kube-api-access-t62nz\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609382    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-service-ca\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609421    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-oauth-serving-cert\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609485    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-trusted-ca-bundle\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609510    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-console-config\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609534    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-oauth-config\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609844 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609585    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-serving-cert\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.609844 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.609620    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f552d66-611f-4118-a361-74964e660f36-tmp\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.610209 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.610181    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-service-ca\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.610399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.610376    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f552d66-611f-4118-a361-74964e660f36-tmp\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.611140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.610717    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-oauth-serving-cert\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.611140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.611086    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-trusted-ca-bundle\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.611720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.611702    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-console-config\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.613155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.613130    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-serving-cert\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.613254 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.613182    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-oauth-config\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.629421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.629392    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62nz\" (UniqueName: \"kubernetes.io/projected/4f552d66-611f-4118-a361-74964e660f36-kube-api-access-t62nz\") pod \"console-78bc879ccd-wpwzs\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") " pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:12.754242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:12.754199    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:13.094796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:13.094766    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bc879ccd-wpwzs"]
Oct 30 19:35:13.097656 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:35:13.097624    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f552d66_611f_4118_a361_74964e660f36.slice/crio-f777b1a8199e17445462db7a2dc16e2ed03dc7c3e6d80a5408f9eb94528bc346 WatchSource:0}: Error finding container f777b1a8199e17445462db7a2dc16e2ed03dc7c3e6d80a5408f9eb94528bc346: Status 404 returned error can't find the container with id f777b1a8199e17445462db7a2dc16e2ed03dc7c3e6d80a5408f9eb94528bc346
Oct 30 19:35:13.805863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:13.805816    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc879ccd-wpwzs" event={"ID":"4f552d66-611f-4118-a361-74964e660f36","Type":"ContainerStarted","Data":"f777b1a8199e17445462db7a2dc16e2ed03dc7c3e6d80a5408f9eb94528bc346"}
Oct 30 19:35:17.731159 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:17.731123    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b665c84d-sdhgp"
Oct 30 19:35:17.824387 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:17.824353    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc879ccd-wpwzs" event={"ID":"4f552d66-611f-4118-a361-74964e660f36","Type":"ContainerStarted","Data":"319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010"}
Oct 30 19:35:17.893541 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:17.893457    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78bc879ccd-wpwzs" podStartSLOduration=1.9363126560000001 podStartE2EDuration="5.893437421s" totalImagesPullingTime="3.957124765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:17.892507948 +0000 UTC m=+42.393211673" watchObservedRunningTime="2026-10-30 19:35:17.893437421 +0000 UTC m=+42.394141174"
Oct 30 19:35:20.834579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.834338    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bc879ccd-wpwzs"]
Oct 30 19:35:20.878618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.878586    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58c7c7bc48-5s4mz"]
Oct 30 19:35:20.901534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.901508    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.908672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.908111    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c7c7bc48-5s4mz"]
Oct 30 19:35:20.991601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991562    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-console-config\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.991601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991605    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-serving-cert\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.991814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991728    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b471b68b-1a02-495e-a056-6f136cf3edf4-tmp\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.991814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991769    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-trusted-ca-bundle\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.991928 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991827    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-oauth-serving-cert\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.991928 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991861    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-service-ca\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.991928 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991897    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mgt\" (UniqueName: \"kubernetes.io/projected/b471b68b-1a02-495e-a056-6f136cf3edf4-kube-api-access-z4mgt\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:20.992020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:20.991942    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-oauth-config\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.092792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092714    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-console-config\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.092792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092753    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-serving-cert\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092805    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b471b68b-1a02-495e-a056-6f136cf3edf4-tmp\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092836    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-trusted-ca-bundle\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092856    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-oauth-serving-cert\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092889    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-service-ca\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092930    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mgt\" (UniqueName: \"kubernetes.io/projected/b471b68b-1a02-495e-a056-6f136cf3edf4-kube-api-access-z4mgt\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.092950    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-oauth-config\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.093276    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b471b68b-1a02-495e-a056-6f136cf3edf4-tmp\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093655 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.093625    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-oauth-serving-cert\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093655 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.093646    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-console-config\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093880 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.093855    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-service-ca\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.093928 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.093911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-trusted-ca-bundle\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.095230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.095206    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-serving-cert\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.095459 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.095437    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-oauth-config\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.114520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.114498    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mgt\" (UniqueName: \"kubernetes.io/projected/b471b68b-1a02-495e-a056-6f136cf3edf4-kube-api-access-z4mgt\") pod \"console-58c7c7bc48-5s4mz\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") " pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.226520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.226488    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:21.544746 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.544716    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c7c7bc48-5s4mz"]
Oct 30 19:35:21.545994 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:35:21.545963    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb471b68b_1a02_495e_a056_6f136cf3edf4.slice/crio-a1a8b41b2f5ba222f82fddd63aed8696eb914a7bc2a08625faee67965b83f971 WatchSource:0}: Error finding container a1a8b41b2f5ba222f82fddd63aed8696eb914a7bc2a08625faee67965b83f971: Status 404 returned error can't find the container with id a1a8b41b2f5ba222f82fddd63aed8696eb914a7bc2a08625faee67965b83f971
Oct 30 19:35:21.840444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.840354    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c7c7bc48-5s4mz" event={"ID":"b471b68b-1a02-495e-a056-6f136cf3edf4","Type":"ContainerStarted","Data":"36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491"}
Oct 30 19:35:21.840444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.840418    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c7c7bc48-5s4mz" event={"ID":"b471b68b-1a02-495e-a056-6f136cf3edf4","Type":"ContainerStarted","Data":"a1a8b41b2f5ba222f82fddd63aed8696eb914a7bc2a08625faee67965b83f971"}
Oct 30 19:35:21.884898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:21.884850    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58c7c7bc48-5s4mz" podStartSLOduration=1.884837488 podStartE2EDuration="1.884837488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:21.883331075 +0000 UTC m=+46.384034799" watchObservedRunningTime="2026-10-30 19:35:21.884837488 +0000 UTC m=+46.385541212"
Oct 30 19:35:22.755328 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:22.755292    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:31.227514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:31.227474    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:31.227996 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:31.227559    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:31.232183 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:31.232158    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:31.879701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:31.879664    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:35:32.220055 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:35:32.220014    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:35:45.853885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:45.853820    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-78bc879ccd-wpwzs" podUID="4f552d66-611f-4118-a361-74964e660f36" containerName="console" containerID="cri-o://319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010" gracePeriod=15
Oct 30 19:35:46.129292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.129227    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:46.194404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194378    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f552d66-611f-4118-a361-74964e660f36-kube-api-access-t62nz\" (UniqueName: \"kubernetes.io/projected/4f552d66-611f-4118-a361-74964e660f36-kube-api-access-t62nz\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194433    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-oauth-serving-cert\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194487    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-console-config\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-console-config\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194517    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f552d66-611f-4118-a361-74964e660f36-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f552d66-611f-4118-a361-74964e660f36-tmp\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194551    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-service-ca\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194774 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194586    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-serving-cert\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194774 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194631    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-oauth-config\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194774 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194668    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-trusted-ca-bundle\") pod \"4f552d66-611f-4118-a361-74964e660f36\" (UID: \"4f552d66-611f-4118-a361-74964e660f36\") "
Oct 30 19:35:46.194944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194909    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f552d66-611f-4118-a361-74964e660f36-tmp" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:46.195072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.194938    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-console-config" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:46.195190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.195167    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-oauth-serving-cert" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:46.195329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.195217    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-trusted-ca-bundle" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:46.195329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.195253    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-service-ca" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:46.196824 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.196801    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f552d66-611f-4118-a361-74964e660f36-kube-api-access-t62nz" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "kube-api-access-t62nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:46.196906 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.196806    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-serving-cert" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:46.196948 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.196899    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-oauth-config" pod "4f552d66-611f-4118-a361-74964e660f36" (UID: "4f552d66-611f-4118-a361-74964e660f36"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:46.295971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.295944    4519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.295971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.295968    4519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-serving-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.296108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.295978    4519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f552d66-611f-4118-a361-74964e660f36-console-oauth-config\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.296108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.295988    4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-trusted-ca-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.296108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.295997    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t62nz\" (UniqueName: \"kubernetes.io/projected/4f552d66-611f-4118-a361-74964e660f36-kube-api-access-t62nz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.296108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.296006    4519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-oauth-serving-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.296108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.296015    4519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f552d66-611f-4118-a361-74964e660f36-console-config\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.296108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.296024    4519 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f552d66-611f-4118-a361-74964e660f36-tmp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:46.354348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.354203    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-644b5886bf-6qghr"]
Oct 30 19:35:46.354528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.354504    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4f552d66-611f-4118-a361-74964e660f36" containerName="console"
Oct 30 19:35:46.354528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.354521    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f552d66-611f-4118-a361-74964e660f36" containerName="console"
Oct 30 19:35:46.354624 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.354586    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f552d66-611f-4118-a361-74964e660f36" containerName="console"
Oct 30 19:35:46.378616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.378576    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.379907 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.379846    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644b5886bf-6qghr"]
Oct 30 19:35:46.497760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.497717    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-service-ca\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.497915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.497794    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-oauth-serving-cert\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.497915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.497829    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-oauth-config\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.497915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.497851    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-tmp\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.498079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.497936    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-serving-cert\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.498079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.497983    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-trusted-ca-bundle\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.498079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.498041    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n7jf\" (UniqueName: \"kubernetes.io/projected/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-kube-api-access-2n7jf\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.498079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.498062    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-config\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599253 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599223    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-oauth-serving-cert\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599266    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-oauth-config\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599296    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-tmp\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599349    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-serving-cert\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599375    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-trusted-ca-bundle\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599395    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n7jf\" (UniqueName: \"kubernetes.io/projected/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-kube-api-access-2n7jf\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599417    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-config\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599453    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-service-ca\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.599792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.599760    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-tmp\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.600030 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.600009    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-oauth-serving-cert\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.600153 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.600131    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-config\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.600297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.600277    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-service-ca\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.600365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.600350    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-trusted-ca-bundle\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.601997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.601975    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-oauth-config\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.605827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.602385    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-console-serving-cert\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.619389 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.619368    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n7jf\" (UniqueName: \"kubernetes.io/projected/f5ef3fe4-5e10-4fa7-9b73-eb5004650f06-kube-api-access-2n7jf\") pod \"console-644b5886bf-6qghr\" (UID: \"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06\") " pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.703577 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.703548    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:46.932057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.932005    4519 generic.go:356] "Generic (PLEG): container finished" podID="4f552d66-611f-4118-a361-74964e660f36" containerID="319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010" exitCode=0
Oct 30 19:35:46.932535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.932068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc879ccd-wpwzs" event={"ID":"4f552d66-611f-4118-a361-74964e660f36","Type":"ContainerDied","Data":"319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010"}
Oct 30 19:35:46.932535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.932079    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc879ccd-wpwzs"
Oct 30 19:35:46.932535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.932123    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc879ccd-wpwzs" event={"ID":"4f552d66-611f-4118-a361-74964e660f36","Type":"ContainerDied","Data":"f777b1a8199e17445462db7a2dc16e2ed03dc7c3e6d80a5408f9eb94528bc346"}
Oct 30 19:35:46.932535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.932214    4519 scope.go:122] "RemoveContainer" containerID="319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010"
Oct 30 19:35:46.940996 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.940981    4519 scope.go:122] "RemoveContainer" containerID="319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010"
Oct 30 19:35:46.941277 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:35:46.941256    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010\": container with ID starting with 319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010 not found: ID does not exist" containerID="319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010"
Oct 30 19:35:46.941353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.941288    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010"} err="failed to get container status \"319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010\": rpc error: code = NotFound desc = could not find container \"319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010\": container with ID starting with 319dbec1f3cab1ef3cde1a683534f25d91ab7971c0bddf48f144da975fd2e010 not found: ID does not exist"
Oct 30 19:35:46.982885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.982831    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bc879ccd-wpwzs"]
Oct 30 19:35:46.994789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:46.994767    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78bc879ccd-wpwzs"]
Oct 30 19:35:47.033197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:47.033174    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644b5886bf-6qghr"]
Oct 30 19:35:47.034648 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:35:47.034624    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ef3fe4_5e10_4fa7_9b73_eb5004650f06.slice/crio-dc95335ff9c468e339d704598dff4db4d6358ac750f13f7989c7b99275d0d3ca WatchSource:0}: Error finding container dc95335ff9c468e339d704598dff4db4d6358ac750f13f7989c7b99275d0d3ca: Status 404 returned error can't find the container with id dc95335ff9c468e339d704598dff4db4d6358ac750f13f7989c7b99275d0d3ca
Oct 30 19:35:47.609866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:47.609832    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f552d66-611f-4118-a361-74964e660f36" path="/var/lib/kubelet/pods/4f552d66-611f-4118-a361-74964e660f36/volumes"
Oct 30 19:35:47.937649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:47.937612    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644b5886bf-6qghr" event={"ID":"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06","Type":"ContainerStarted","Data":"d7aa1870a9a54c2cc5ab4e0c79d2484b37c73322a039f6a3241cae38830c9c0b"}
Oct 30 19:35:47.938025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:47.937668    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644b5886bf-6qghr" event={"ID":"f5ef3fe4-5e10-4fa7-9b73-eb5004650f06","Type":"ContainerStarted","Data":"dc95335ff9c468e339d704598dff4db4d6358ac750f13f7989c7b99275d0d3ca"}
Oct 30 19:35:47.978318 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:47.978165    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-644b5886bf-6qghr" podStartSLOduration=1.978148268 podStartE2EDuration="1.978148268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:47.977510703 +0000 UTC m=+72.478214429" watchObservedRunningTime="2026-10-30 19:35:47.978148268 +0000 UTC m=+72.478851992"
Oct 30 19:35:56.703840 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:56.703806    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:56.704316 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:56.703880    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:56.708386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:56.708356    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:56.975425 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:56.975347    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-644b5886bf-6qghr"
Oct 30 19:35:57.046667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:57.046635    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c7c7bc48-5s4mz"]
Oct 30 19:35:57.976652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:57.976619    4519 generic.go:356] "Generic (PLEG): container finished" podID="35623b34-ad3c-4170-99a2-76ff61156c80" containerID="8d1d82e6a1527cdb0f5a9a2593b96ec33bfed205b606b4e3f8318f9d3da50181" exitCode=0
Oct 30 19:35:57.977007 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:57.976690    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h2cdk-f7vp9" event={"ID":"35623b34-ad3c-4170-99a2-76ff61156c80","Type":"ContainerDied","Data":"8d1d82e6a1527cdb0f5a9a2593b96ec33bfed205b606b4e3f8318f9d3da50181"}
Oct 30 19:35:59.109976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.109954    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:35:59.206166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206132    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\") pod \"35623b34-ad3c-4170-99a2-76ff61156c80\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") "
Oct 30 19:35:59.206320 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206197    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\" (UniqueName: \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\") pod \"35623b34-ad3c-4170-99a2-76ff61156c80\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") "
Oct 30 19:35:59.206320 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206220    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\") pod \"35623b34-ad3c-4170-99a2-76ff61156c80\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") "
Oct 30 19:35:59.206434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206330    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\") pod \"35623b34-ad3c-4170-99a2-76ff61156c80\" (UID: \"35623b34-ad3c-4170-99a2-76ff61156c80\") "
Oct 30 19:35:59.206559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206541    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle" pod "35623b34-ad3c-4170-99a2-76ff61156c80" (UID: "35623b34-ad3c-4170-99a2-76ff61156c80"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:59.206607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206550    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle" pod "35623b34-ad3c-4170-99a2-76ff61156c80" (UID: "35623b34-ad3c-4170-99a2-76ff61156c80"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:59.206649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206607    4519 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-service-ca-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:59.206649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206618    4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35623b34-ad3c-4170-99a2-76ff61156c80-trusted-ca-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:59.206728 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.206653    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path" pod "35623b34-ad3c-4170-99a2-76ff61156c80" (UID: "35623b34-ad3c-4170-99a2-76ff61156c80"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:59.208475 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.208444    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs" pod "35623b34-ad3c-4170-99a2-76ff61156c80" (UID: "35623b34-ad3c-4170-99a2-76ff61156c80"). InnerVolumeSpecName "kube-api-access-zhjrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:59.307984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.307900    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhjrs\" (UniqueName: \"kubernetes.io/projected/35623b34-ad3c-4170-99a2-76ff61156c80-kube-api-access-zhjrs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:59.307984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.307931    4519 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/35623b34-ad3c-4170-99a2-76ff61156c80-archives-path\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:35:59.984686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.984659    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2cdk-f7vp9"
Oct 30 19:35:59.984857 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.984663    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h2cdk-f7vp9" event={"ID":"35623b34-ad3c-4170-99a2-76ff61156c80","Type":"ContainerDied","Data":"c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09"}
Oct 30 19:35:59.984857 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:35:59.984779    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8979816e3d3c4ef4a1a97353c9d7aa276ce9234662e0d94ca5dd91b80714d09"
Oct 30 19:36:02.218636 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:36:02.218593    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:36:06.271152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.271116    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jlbbw-59v2g"]
Oct 30 19:36:06.271647 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.271623    4519 cpu_manager.go:414] "containerMap: removing container" podUID="35623b34-ad3c-4170-99a2-76ff61156c80" containerName="insights-gathering"
Oct 30 19:36:06.271724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.271646    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35623b34-ad3c-4170-99a2-76ff61156c80" containerName="insights-gathering"
Oct 30 19:36:06.271775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.271742    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="35623b34-ad3c-4170-99a2-76ff61156c80" containerName="insights-gathering"
Oct 30 19:36:06.274366 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.274344    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.301584 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.301562    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hmkn8\""
Oct 30 19:36:06.301696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.301601    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:36:06.328586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.328551    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jlbbw-59v2g"]
Oct 30 19:36:06.328772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.328637    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:36:06.355482 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.355440    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-trusted-ca-bundle\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.355594 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.355496    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8m6j\" (UniqueName: \"kubernetes.io/projected/ee260905-a174-4529-95c0-f900b63d47ad-kube-api-access-s8m6j\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.355594 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.355519    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-service-ca-bundle\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.355696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.355620    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee260905-a174-4529-95c0-f900b63d47ad-archives-path\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.455986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.455957    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8m6j\" (UniqueName: \"kubernetes.io/projected/ee260905-a174-4529-95c0-f900b63d47ad-kube-api-access-s8m6j\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.456113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.456000    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-service-ca-bundle\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.456113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.456040    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee260905-a174-4529-95c0-f900b63d47ad-archives-path\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.456113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.456092    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-trusted-ca-bundle\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.456558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.456534    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee260905-a174-4529-95c0-f900b63d47ad-archives-path\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.456805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.456787    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-service-ca-bundle\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.456888 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.456870    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-trusted-ca-bundle\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.477919 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.477894    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8m6j\" (UniqueName: \"kubernetes.io/projected/ee260905-a174-4529-95c0-f900b63d47ad-kube-api-access-s8m6j\") pod \"periodic-gathering-jlbbw-59v2g\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") " pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.584650 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.584571    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:36:06.878912 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:06.878885    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jlbbw-59v2g"]
Oct 30 19:36:06.880429 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:36:06.880401    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee260905_a174_4529_95c0_f900b63d47ad.slice/crio-ac1b9f3dfcd584f1a1f72bfbc9fddc5a82d36c7cc798fdd4fc160dc1bee9a54e WatchSource:0}: Error finding container ac1b9f3dfcd584f1a1f72bfbc9fddc5a82d36c7cc798fdd4fc160dc1bee9a54e: Status 404 returned error can't find the container with id ac1b9f3dfcd584f1a1f72bfbc9fddc5a82d36c7cc798fdd4fc160dc1bee9a54e
Oct 30 19:36:07.009243 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:07.009198    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlbbw-59v2g" event={"ID":"ee260905-a174-4529-95c0-f900b63d47ad","Type":"ContainerStarted","Data":"0d639d41d7fbc799b2d5c12499b599df93d9479d4b7d51c5ece94b5617b36e7b"}
Oct 30 19:36:07.009422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:07.009261    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlbbw-59v2g" event={"ID":"ee260905-a174-4529-95c0-f900b63d47ad","Type":"ContainerStarted","Data":"ac1b9f3dfcd584f1a1f72bfbc9fddc5a82d36c7cc798fdd4fc160dc1bee9a54e"}
Oct 30 19:36:07.047958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:07.047901    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jlbbw-59v2g" podStartSLOduration=1.047885957 podStartE2EDuration="1.047885957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:07.047650903 +0000 UTC m=+91.548354627" watchObservedRunningTime="2026-10-30 19:36:07.047885957 +0000 UTC m=+91.548589681"
Oct 30 19:36:22.066427 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.066376    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58c7c7bc48-5s4mz" podUID="b471b68b-1a02-495e-a056-6f136cf3edf4" containerName="console" containerID="cri-o://36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491" gracePeriod=15
Oct 30 19:36:22.316266 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.316235    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:36:22.370299 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370214    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b471b68b-1a02-495e-a056-6f136cf3edf4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b471b68b-1a02-495e-a056-6f136cf3edf4-tmp\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370299 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370261    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-service-ca\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-service-ca\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370345    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-serving-cert\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370376    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-trusted-ca-bundle\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370409    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-oauth-config\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370447    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-console-config\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-console-config\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370531    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-oauth-serving-cert\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370546    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b471b68b-1a02-495e-a056-6f136cf3edf4-tmp" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:22.370601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370582    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b471b68b-1a02-495e-a056-6f136cf3edf4-kube-api-access-z4mgt\" (UniqueName: \"kubernetes.io/projected/b471b68b-1a02-495e-a056-6f136cf3edf4-kube-api-access-z4mgt\") pod \"b471b68b-1a02-495e-a056-6f136cf3edf4\" (UID: \"b471b68b-1a02-495e-a056-6f136cf3edf4\") "
Oct 30 19:36:22.370935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370667    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-service-ca" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:22.370935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370840    4519 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b471b68b-1a02-495e-a056-6f136cf3edf4-tmp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.370935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370857    4519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.370935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.370909    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-console-config" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:22.371237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.371209    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-trusted-ca-bundle" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:22.371303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.371220    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-oauth-serving-cert" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:22.373017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.372982    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-serving-cert" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:22.373017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.373000    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-oauth-config" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:22.373189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.373038    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b471b68b-1a02-495e-a056-6f136cf3edf4-kube-api-access-z4mgt" pod "b471b68b-1a02-495e-a056-6f136cf3edf4" (UID: "b471b68b-1a02-495e-a056-6f136cf3edf4"). InnerVolumeSpecName "kube-api-access-z4mgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:22.471567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.471527    4519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-serving-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.471567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.471558    4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-trusted-ca-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.471567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.471571    4519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b471b68b-1a02-495e-a056-6f136cf3edf4-console-oauth-config\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.471796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.471580    4519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-console-config\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.471796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.471589    4519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b471b68b-1a02-495e-a056-6f136cf3edf4-oauth-serving-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.471796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:22.471597    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4mgt\" (UniqueName: \"kubernetes.io/projected/b471b68b-1a02-495e-a056-6f136cf3edf4-kube-api-access-z4mgt\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:36:23.065030 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.064992    4519 generic.go:356] "Generic (PLEG): container finished" podID="b471b68b-1a02-495e-a056-6f136cf3edf4" containerID="36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491" exitCode=0
Oct 30 19:36:23.065170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.065034    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c7c7bc48-5s4mz" event={"ID":"b471b68b-1a02-495e-a056-6f136cf3edf4","Type":"ContainerDied","Data":"36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491"}
Oct 30 19:36:23.065170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.065057    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c7c7bc48-5s4mz"
Oct 30 19:36:23.065170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.065098    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c7c7bc48-5s4mz" event={"ID":"b471b68b-1a02-495e-a056-6f136cf3edf4","Type":"ContainerDied","Data":"a1a8b41b2f5ba222f82fddd63aed8696eb914a7bc2a08625faee67965b83f971"}
Oct 30 19:36:23.065170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.065122    4519 scope.go:122] "RemoveContainer" containerID="36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491"
Oct 30 19:36:23.073443 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.073279    4519 scope.go:122] "RemoveContainer" containerID="36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491"
Oct 30 19:36:23.073713 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:36:23.073563    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491\": container with ID starting with 36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491 not found: ID does not exist" containerID="36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491"
Oct 30 19:36:23.073713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.073591    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491"} err="failed to get container status \"36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491\": rpc error: code = NotFound desc = could not find container \"36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491\": container with ID starting with 36c96bcdf4fae4aa06c69b065515434aa2c98faa835a8627f26618ea16d8e491 not found: ID does not exist"
Oct 30 19:36:23.116942 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.116915    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c7c7bc48-5s4mz"]
Oct 30 19:36:23.128921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.128901    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58c7c7bc48-5s4mz"]
Oct 30 19:36:23.609669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:36:23.609637    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b471b68b-1a02-495e-a056-6f136cf3edf4" path="/var/lib/kubelet/pods/b471b68b-1a02-495e-a056-6f136cf3edf4/volumes"
Oct 30 19:36:32.219124 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:36:32.219086    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:37:02.221779 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:37:02.221739    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:37:31.285933 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:31.285902    4519 generic.go:356] "Generic (PLEG): container finished" podID="ee260905-a174-4529-95c0-f900b63d47ad" containerID="0d639d41d7fbc799b2d5c12499b599df93d9479d4b7d51c5ece94b5617b36e7b" exitCode=0
Oct 30 19:37:31.286329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:31.285977    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlbbw-59v2g" event={"ID":"ee260905-a174-4529-95c0-f900b63d47ad","Type":"ContainerDied","Data":"0d639d41d7fbc799b2d5c12499b599df93d9479d4b7d51c5ece94b5617b36e7b"}
Oct 30 19:37:32.220426 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:37:32.220390    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:37:32.430020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.429991    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:37:32.484820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.484730    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-service-ca-bundle\") pod \"ee260905-a174-4529-95c0-f900b63d47ad\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") "
Oct 30 19:37:32.484820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.484777    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee260905-a174-4529-95c0-f900b63d47ad-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee260905-a174-4529-95c0-f900b63d47ad-archives-path\") pod \"ee260905-a174-4529-95c0-f900b63d47ad\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") "
Oct 30 19:37:32.484820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.484817    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-trusted-ca-bundle\") pod \"ee260905-a174-4529-95c0-f900b63d47ad\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") "
Oct 30 19:37:32.485107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.484850    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee260905-a174-4529-95c0-f900b63d47ad-kube-api-access-s8m6j\" (UniqueName: \"kubernetes.io/projected/ee260905-a174-4529-95c0-f900b63d47ad-kube-api-access-s8m6j\") pod \"ee260905-a174-4529-95c0-f900b63d47ad\" (UID: \"ee260905-a174-4529-95c0-f900b63d47ad\") "
Oct 30 19:37:32.485173 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.485136    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-service-ca-bundle" pod "ee260905-a174-4529-95c0-f900b63d47ad" (UID: "ee260905-a174-4529-95c0-f900b63d47ad"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:32.485230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.485205    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee260905-a174-4529-95c0-f900b63d47ad-archives-path" pod "ee260905-a174-4529-95c0-f900b63d47ad" (UID: "ee260905-a174-4529-95c0-f900b63d47ad"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:32.485230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.485217    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-trusted-ca-bundle" pod "ee260905-a174-4529-95c0-f900b63d47ad" (UID: "ee260905-a174-4529-95c0-f900b63d47ad"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:32.487099 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.487065    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee260905-a174-4529-95c0-f900b63d47ad-kube-api-access-s8m6j" pod "ee260905-a174-4529-95c0-f900b63d47ad" (UID: "ee260905-a174-4529-95c0-f900b63d47ad"). InnerVolumeSpecName "kube-api-access-s8m6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:32.585750 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.585712    4519 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-service-ca-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:37:32.585750 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.585745    4519 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee260905-a174-4529-95c0-f900b63d47ad-archives-path\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:37:32.585750 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.585756    4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee260905-a174-4529-95c0-f900b63d47ad-trusted-ca-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:37:32.585995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:32.585764    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8m6j\" (UniqueName: \"kubernetes.io/projected/ee260905-a174-4529-95c0-f900b63d47ad-kube-api-access-s8m6j\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:37:33.293934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:33.293898    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlbbw-59v2g" event={"ID":"ee260905-a174-4529-95c0-f900b63d47ad","Type":"ContainerDied","Data":"ac1b9f3dfcd584f1a1f72bfbc9fddc5a82d36c7cc798fdd4fc160dc1bee9a54e"}
Oct 30 19:37:33.294185 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:33.293948    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac1b9f3dfcd584f1a1f72bfbc9fddc5a82d36c7cc798fdd4fc160dc1bee9a54e"
Oct 30 19:37:33.294185 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:37:33.293914    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlbbw-59v2g"
Oct 30 19:38:02.221071 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:38:02.221029    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:38:32.221036 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:38:32.220958    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:39:02.220582 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:39:02.220538    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:39:32.219636 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:39:32.219591    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:40:02.220354 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:40:02.220312    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:40:30.729268 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.729203    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"]
Oct 30 19:40:30.729866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.729586    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b471b68b-1a02-495e-a056-6f136cf3edf4" containerName="console"
Oct 30 19:40:30.729866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.729605    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b471b68b-1a02-495e-a056-6f136cf3edf4" containerName="console"
Oct 30 19:40:30.729866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.729618    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ee260905-a174-4529-95c0-f900b63d47ad" containerName="insights-gathering"
Oct 30 19:40:30.729866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.729623    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee260905-a174-4529-95c0-f900b63d47ad" containerName="insights-gathering"
Oct 30 19:40:30.729866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.729687    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b471b68b-1a02-495e-a056-6f136cf3edf4" containerName="console"
Oct 30 19:40:30.729866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.729696    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee260905-a174-4529-95c0-f900b63d47ad" containerName="insights-gathering"
Oct 30 19:40:30.732784 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.732765    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.759587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.759541    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:40:30.759587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.759542    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:40:30.759793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.759630    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-4crz9\""
Oct 30 19:40:30.771515 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.771459    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"]
Oct 30 19:40:30.886298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.886259    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lkt\" (UniqueName: \"kubernetes.io/projected/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-kube-api-access-d6lkt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.886298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.886303    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.886584 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.886326    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.934623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.934586    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"]
Oct 30 19:40:30.938047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.938020    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:30.958837 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.958799    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"]
Oct 30 19:40:30.987414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.987308    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lkt\" (UniqueName: \"kubernetes.io/projected/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-kube-api-access-d6lkt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.987414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.987356    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.987673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.987523    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.987748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.987733    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:30.987868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:30.987851    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:31.003198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.003170    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-kx686\""
Oct 30 19:40:31.006625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.006603    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lkt\" (UniqueName: \"kubernetes.io/projected/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-kube-api-access-d6lkt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:31.048616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.048579    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:40:31.088648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.088275    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbc49\" (UniqueName: \"kubernetes.io/projected/97b0c2a7-60c7-4431-8d90-e9dd523cf411-kube-api-access-lbc49\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.088648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.088334    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.088648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.088376    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.189265 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.189224    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbc49\" (UniqueName: \"kubernetes.io/projected/97b0c2a7-60c7-4431-8d90-e9dd523cf411-kube-api-access-lbc49\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.189265 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.189268    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.189556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.189289    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.189678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.189661    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.189753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.189728    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.209350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.209316    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbc49\" (UniqueName: \"kubernetes.io/projected/97b0c2a7-60c7-4431-8d90-e9dd523cf411-kube-api-access-lbc49\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.254190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.254092    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:40:31.374753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.374724    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"]
Oct 30 19:40:31.376793 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:31.376758    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb69e79d_0af3_45b2_b4b5_5b24d0933e86.slice/crio-1d89809d9d530d079c486178e7c224e9427c4f4fb18f1c0b8ccee494917f498b WatchSource:0}: Error finding container 1d89809d9d530d079c486178e7c224e9427c4f4fb18f1c0b8ccee494917f498b: Status 404 returned error can't find the container with id 1d89809d9d530d079c486178e7c224e9427c4f4fb18f1c0b8ccee494917f498b
Oct 30 19:40:31.378649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.378632    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:40:31.555535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.555504    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"]
Oct 30 19:40:31.556618 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:31.556590    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b0c2a7_60c7_4431_8d90_e9dd523cf411.slice/crio-43bf1933f62b03c5b1d69fa0927f7e4f73c817b8043cf5235968ad55b6486862 WatchSource:0}: Error finding container 43bf1933f62b03c5b1d69fa0927f7e4f73c817b8043cf5235968ad55b6486862: Status 404 returned error can't find the container with id 43bf1933f62b03c5b1d69fa0927f7e4f73c817b8043cf5235968ad55b6486862
Oct 30 19:40:31.880127 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.880033    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk" event={"ID":"bb69e79d-0af3-45b2-b4b5-5b24d0933e86","Type":"ContainerStarted","Data":"1d89809d9d530d079c486178e7c224e9427c4f4fb18f1c0b8ccee494917f498b"}
Oct 30 19:40:31.881125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:31.881098    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7" event={"ID":"97b0c2a7-60c7-4431-8d90-e9dd523cf411","Type":"ContainerStarted","Data":"43bf1933f62b03c5b1d69fa0927f7e4f73c817b8043cf5235968ad55b6486862"}
Oct 30 19:40:32.229332 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:40:32.229245    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:40:34.122597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.122563    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"]
Oct 30 19:40:34.126096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.126070    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.153140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.153107    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-sqg5j\""
Oct 30 19:40:34.154532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.154501    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"]
Oct 30 19:40:34.218013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.217971    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dkhg\" (UniqueName: \"kubernetes.io/projected/3f33a2ae-592e-4750-be1d-827ea1b8d175-kube-api-access-2dkhg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.218013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.218017    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.218234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.218047    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.319174 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.319135    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dkhg\" (UniqueName: \"kubernetes.io/projected/3f33a2ae-592e-4750-be1d-827ea1b8d175-kube-api-access-2dkhg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.319382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.319187    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.319382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.319220    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.319728 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.319701    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.319798 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.319726    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.340181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.340108    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dkhg\" (UniqueName: \"kubernetes.io/projected/3f33a2ae-592e-4750-be1d-827ea1b8d175-kube-api-access-2dkhg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.446673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.446634    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:40:34.759362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.759276    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"]
Oct 30 19:40:34.762716 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:34.762683    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f33a2ae_592e_4750_be1d_827ea1b8d175.slice/crio-d6e8dc2d1f8c94cc5a7b7cd0573f6ccaa436506ffa702b496418dbdc34fce486 WatchSource:0}: Error finding container d6e8dc2d1f8c94cc5a7b7cd0573f6ccaa436506ffa702b496418dbdc34fce486: Status 404 returned error can't find the container with id d6e8dc2d1f8c94cc5a7b7cd0573f6ccaa436506ffa702b496418dbdc34fce486
Oct 30 19:40:34.894144 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:34.894105    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m" event={"ID":"3f33a2ae-592e-4750-be1d-827ea1b8d175","Type":"ContainerStarted","Data":"d6e8dc2d1f8c94cc5a7b7cd0573f6ccaa436506ffa702b496418dbdc34fce486"}
Oct 30 19:40:35.391347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.391238    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"]
Oct 30 19:40:35.395174 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.395128    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.414096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.414064    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-hdv8p\""
Oct 30 19:40:35.414842 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.414805    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"]
Oct 30 19:40:35.495165 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.495128    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"]
Oct 30 19:40:35.498945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.498912    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.513775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.513737    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-fq97p\""
Oct 30 19:40:35.519608 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.519571    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"]
Oct 30 19:40:35.533544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.533507    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.533709 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.533568    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.533810 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.533731    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8z6t\" (UniqueName: \"kubernetes.io/projected/d5c2aa8a-87b2-406c-81a1-069e66f8f059-kube-api-access-j8z6t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.589511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.589456    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"]
Oct 30 19:40:35.594586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.594556    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.611767 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.611733    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-h757x\""
Oct 30 19:40:35.613865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.613787    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"]
Oct 30 19:40:35.634907 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.634876    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.635084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.634920    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8z6t\" (UniqueName: \"kubernetes.io/projected/d5c2aa8a-87b2-406c-81a1-069e66f8f059-kube-api-access-j8z6t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.635084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.635005    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.635084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.635040    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.635084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.635061    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bg2w\" (UniqueName: \"kubernetes.io/projected/702acfe9-e564-4066-840f-a2b4043f8d34-kube-api-access-4bg2w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.635219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.635116    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.635447 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.635424    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.635614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.635502    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.655111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.655037    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8z6t\" (UniqueName: \"kubernetes.io/projected/d5c2aa8a-87b2-406c-81a1-069e66f8f059-kube-api-access-j8z6t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.701679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.701635    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"]
Oct 30 19:40:35.705987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.705955    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.726289 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.724815    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-c6dkh\""
Oct 30 19:40:35.728493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.728440    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"]
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.736657    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.736740    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.736772    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bg2w\" (UniqueName: \"kubernetes.io/projected/702acfe9-e564-4066-840f-a2b4043f8d34-kube-api-access-4bg2w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.736830    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.736864    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdf76\" (UniqueName: \"kubernetes.io/projected/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-kube-api-access-fdf76\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.736909    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.737280    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.737870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.737534    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.741421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.741391    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-hdv8p\""
Oct 30 19:40:35.748677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.748634    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:40:35.758094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.758054    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bg2w\" (UniqueName: \"kubernetes.io/projected/702acfe9-e564-4066-840f-a2b4043f8d34-kube-api-access-4bg2w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.838417    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.838485    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2r4\" (UniqueName: \"kubernetes.io/projected/9d872bd0-61a4-454c-a4a3-150dac29fe49-kube-api-access-2h2r4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.838520    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdf76\" (UniqueName: \"kubernetes.io/projected/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-kube-api-access-fdf76\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.838565    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.838594    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.838632    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.838986    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.839248    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.845502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.841415    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-fq97p\""
Oct 30 19:40:35.848587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.848555    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:40:35.864832 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.859607    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdf76\" (UniqueName: \"kubernetes.io/projected/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-kube-api-access-fdf76\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.924742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.924705    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:40:35.939726 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.939686    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.939889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.939748    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.939889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.939824    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2r4\" (UniqueName: \"kubernetes.io/projected/9d872bd0-61a4-454c-a4a3-150dac29fe49-kube-api-access-2h2r4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.940523 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.940497    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.940626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.940502    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:35.969516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:35.967666    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2r4\" (UniqueName: \"kubernetes.io/projected/9d872bd0-61a4-454c-a4a3-150dac29fe49-kube-api-access-2h2r4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:36.028843 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.028801    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:40:36.071722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.068366    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"]
Oct 30 19:40:36.071722 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:36.070304    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c2aa8a_87b2_406c_81a1_069e66f8f059.slice/crio-985fda29aed7001ff418cdc1b191f6df2515699033e9723c4b44e2e2ae8a8378 WatchSource:0}: Error finding container 985fda29aed7001ff418cdc1b191f6df2515699033e9723c4b44e2e2ae8a8378: Status 404 returned error can't find the container with id 985fda29aed7001ff418cdc1b191f6df2515699033e9723c4b44e2e2ae8a8378
Oct 30 19:40:36.173810 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.173775    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"]
Oct 30 19:40:36.175206 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:36.175144    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702acfe9_e564_4066_840f_a2b4043f8d34.slice/crio-a1b9a8ba6de31dbf3b291044a6fcbec2dc36a3cbdb046a53cc578beb7a77581b WatchSource:0}: Error finding container a1b9a8ba6de31dbf3b291044a6fcbec2dc36a3cbdb046a53cc578beb7a77581b: Status 404 returned error can't find the container with id a1b9a8ba6de31dbf3b291044a6fcbec2dc36a3cbdb046a53cc578beb7a77581b
Oct 30 19:40:36.257646 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.257608    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"]
Oct 30 19:40:36.259069 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:36.259037    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c95a41_ea0a_4052_8cee_eb113cc9fb3c.slice/crio-7f5e15dd408feceecf2eb126651f3fb6339457494346c4e85578aff0f74cf19f WatchSource:0}: Error finding container 7f5e15dd408feceecf2eb126651f3fb6339457494346c4e85578aff0f74cf19f: Status 404 returned error can't find the container with id 7f5e15dd408feceecf2eb126651f3fb6339457494346c4e85578aff0f74cf19f
Oct 30 19:40:36.357754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.357718    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"]
Oct 30 19:40:36.361550 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:36.361506    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d872bd0_61a4_454c_a4a3_150dac29fe49.slice/crio-9ee4c6d701493317664ac860e19f09ead2f41c50b39a9206f347a1517d98f204 WatchSource:0}: Error finding container 9ee4c6d701493317664ac860e19f09ead2f41c50b39a9206f347a1517d98f204: Status 404 returned error can't find the container with id 9ee4c6d701493317664ac860e19f09ead2f41c50b39a9206f347a1517d98f204
Oct 30 19:40:36.564560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.564069    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"]
Oct 30 19:40:36.567936 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.567907    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.585896 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.585867    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:40:36.586810 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.586783    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-94wtv\""
Oct 30 19:40:36.586945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.586913    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:40:36.587019 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.586995    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:40:36.593698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.593585    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"]
Oct 30 19:40:36.648267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648203    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bd04d-c63b-441d-a17b-fff0e36df522-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648360    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648397    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrn5n\" (UniqueName: \"kubernetes.io/projected/924bd04d-c63b-441d-a17b-fff0e36df522-kube-api-access-rrn5n\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648490    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924bd04d-c63b-441d-a17b-fff0e36df522-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648532    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648579    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bd04d-c63b-441d-a17b-fff0e36df522-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648612    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648629    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.648885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.648646    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.694022 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.693980    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"]
Oct 30 19:40:36.697970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.697940    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.720345 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.716680    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-tncb4\""
Oct 30 19:40:36.721163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.721065    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"]
Oct 30 19:40:36.750039 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.749998    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bd04d-c63b-441d-a17b-fff0e36df522-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750058    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9nl9\" (UniqueName: \"kubernetes.io/projected/cdd067e2-0384-46e0-b85f-4580b7a107fe-kube-api-access-x9nl9\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750121    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750149    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750186    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750264    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd067e2-0384-46e0-b85f-4580b7a107fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750308    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bd04d-c63b-441d-a17b-fff0e36df522-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750341    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750364    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750412    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrn5n\" (UniqueName: \"kubernetes.io/projected/924bd04d-c63b-441d-a17b-fff0e36df522-kube-api-access-rrn5n\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750511    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924bd04d-c63b-441d-a17b-fff0e36df522-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750540    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750577    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.750773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750608    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750639    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.750773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.750681    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.752712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.751584    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bd04d-c63b-441d-a17b-fff0e36df522-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.752712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.751932    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.752712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.752170    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.752961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.752787    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.753803 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.753774    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.756234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.756184    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bd04d-c63b-441d-a17b-fff0e36df522-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.766627 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.766591    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bd04d-c63b-441d-a17b-fff0e36df522-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.782712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.782659    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrn5n\" (UniqueName: \"kubernetes.io/projected/924bd04d-c63b-441d-a17b-fff0e36df522-kube-api-access-rrn5n\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.789837 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.789763    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924bd04d-c63b-441d-a17b-fff0e36df522-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-7vbs7\" (UID: \"924bd04d-c63b-441d-a17b-fff0e36df522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.851742    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.851802    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd067e2-0384-46e0-b85f-4580b7a107fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.851842    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.851871    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.851922    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.851953    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.851983    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.852022    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.852072    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9nl9\" (UniqueName: \"kubernetes.io/projected/cdd067e2-0384-46e0-b85f-4580b7a107fe-kube-api-access-x9nl9\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.852880    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.853527    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd067e2-0384-46e0-b85f-4580b7a107fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.853831    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.855014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.854072    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.856132 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.856068    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.861284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.861198    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.863063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.863031    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.878136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.878056    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9nl9\" (UniqueName: \"kubernetes.io/projected/cdd067e2-0384-46e0-b85f-4580b7a107fe-kube-api-access-x9nl9\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.881474 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.881422    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd067e2-0384-46e0-b85f-4580b7a107fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22\" (UID: \"cdd067e2-0384-46e0-b85f-4580b7a107fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:36.895443 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.895063    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:36.939510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.935808    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8" event={"ID":"d5c2aa8a-87b2-406c-81a1-069e66f8f059","Type":"ContainerStarted","Data":"985fda29aed7001ff418cdc1b191f6df2515699033e9723c4b44e2e2ae8a8378"}
Oct 30 19:40:36.943098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.943050    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt" event={"ID":"702acfe9-e564-4066-840f-a2b4043f8d34","Type":"ContainerStarted","Data":"a1b9a8ba6de31dbf3b291044a6fcbec2dc36a3cbdb046a53cc578beb7a77581b"}
Oct 30 19:40:36.946481 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.946359    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g" event={"ID":"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c","Type":"ContainerStarted","Data":"7f5e15dd408feceecf2eb126651f3fb6339457494346c4e85578aff0f74cf19f"}
Oct 30 19:40:36.949461 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:36.949376    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz" event={"ID":"9d872bd0-61a4-454c-a4a3-150dac29fe49","Type":"ContainerStarted","Data":"9ee4c6d701493317664ac860e19f09ead2f41c50b39a9206f347a1517d98f204"}
Oct 30 19:40:37.025247 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.025213    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:37.251313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.251180    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"]
Oct 30 19:40:37.256016 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:37.255975    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924bd04d_c63b_441d_a17b_fff0e36df522.slice/crio-de638b2d6b5d2dc30d524edb1774765b1b24635b8a0f1c22034c0cd234bc2b5f WatchSource:0}: Error finding container de638b2d6b5d2dc30d524edb1774765b1b24635b8a0f1c22034c0cd234bc2b5f: Status 404 returned error can't find the container with id de638b2d6b5d2dc30d524edb1774765b1b24635b8a0f1c22034c0cd234bc2b5f
Oct 30 19:40:37.396537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.396418    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"]
Oct 30 19:40:37.399257 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:37.399215    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd067e2_0384_46e0_b85f_4580b7a107fe.slice/crio-a5a3d48acb7262e50de1e8574f990b63765308fd7aa09b1a90df57c917b5d1de WatchSource:0}: Error finding container a5a3d48acb7262e50de1e8574f990b63765308fd7aa09b1a90df57c917b5d1de: Status 404 returned error can't find the container with id a5a3d48acb7262e50de1e8574f990b63765308fd7aa09b1a90df57c917b5d1de
Oct 30 19:40:37.517115 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.515292    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"]
Oct 30 19:40:37.522062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.521531    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.543348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.543301    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"]
Oct 30 19:40:37.558760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.558721    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vr9\" (UniqueName: \"kubernetes.io/projected/1301a737-4486-40b5-99cb-f95577858354-kube-api-access-68vr9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.558941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.558794    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.558941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.558855    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.661298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.660062    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.661298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.660131    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.661298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.660211    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vr9\" (UniqueName: \"kubernetes.io/projected/1301a737-4486-40b5-99cb-f95577858354-kube-api-access-68vr9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.661298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.660980    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.661298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.661220    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.682402 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.682334    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vr9\" (UniqueName: \"kubernetes.io/projected/1301a737-4486-40b5-99cb-f95577858354-kube-api-access-68vr9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.845727 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.845450    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:40:37.963519 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.963411    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22" event={"ID":"cdd067e2-0384-46e0-b85f-4580b7a107fe","Type":"ContainerStarted","Data":"a5a3d48acb7262e50de1e8574f990b63765308fd7aa09b1a90df57c917b5d1de"}
Oct 30 19:40:37.967436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:37.967380    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" event={"ID":"924bd04d-c63b-441d-a17b-fff0e36df522","Type":"ContainerStarted","Data":"de638b2d6b5d2dc30d524edb1774765b1b24635b8a0f1c22034c0cd234bc2b5f"}
Oct 30 19:40:38.114618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.114578    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"]
Oct 30 19:40:38.118661 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.118630    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.138705 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.138671    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"]
Oct 30 19:40:38.165814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.165556    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkm68\" (UniqueName: \"kubernetes.io/projected/12a0903b-c5ef-4b60-888a-dfcd12f86432-kube-api-access-vkm68\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.165814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.165624    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.165814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.165665    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.232009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.231894    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"]
Oct 30 19:40:38.266944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.266895    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.267122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.266968    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.267122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.267081    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkm68\" (UniqueName: \"kubernetes.io/projected/12a0903b-c5ef-4b60-888a-dfcd12f86432-kube-api-access-vkm68\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.267890 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.267862    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.268160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.268138    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.300609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.300571    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkm68\" (UniqueName: \"kubernetes.io/projected/12a0903b-c5ef-4b60-888a-dfcd12f86432-kube-api-access-vkm68\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.450450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.450411    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:40:38.797742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.797659    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"]
Oct 30 19:40:38.812522 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:40:38.807015    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a0903b_c5ef_4b60_888a_dfcd12f86432.slice/crio-7a9acc22d2f91caf9fafa5653db7a9a709ae9bedf64a4ddce01b0b5e8868b326 WatchSource:0}: Error finding container 7a9acc22d2f91caf9fafa5653db7a9a709ae9bedf64a4ddce01b0b5e8868b326: Status 404 returned error can't find the container with id 7a9acc22d2f91caf9fafa5653db7a9a709ae9bedf64a4ddce01b0b5e8868b326
Oct 30 19:40:38.999169 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:38.999119    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv" event={"ID":"12a0903b-c5ef-4b60-888a-dfcd12f86432","Type":"ContainerStarted","Data":"7a9acc22d2f91caf9fafa5653db7a9a709ae9bedf64a4ddce01b0b5e8868b326"}
Oct 30 19:40:39.016394 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:39.016347    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2" event={"ID":"1301a737-4486-40b5-99cb-f95577858354","Type":"ContainerStarted","Data":"df037babc5508d4fd80b04e185228a4e5444ee7a8d3f67b49aa65e144e7cf52d"}
Oct 30 19:40:45.076676 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:45.076634    4519 generic.go:356] "Generic (PLEG): container finished" podID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerID="8533d814f5b95c00b86b28d99fe84ebaf506c3380146f32babdcb012ab9729b4" exitCode=0
Oct 30 19:40:45.077132 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:45.076767    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk" event={"ID":"bb69e79d-0af3-45b2-b4b5-5b24d0933e86","Type":"ContainerDied","Data":"8533d814f5b95c00b86b28d99fe84ebaf506c3380146f32babdcb012ab9729b4"}
Oct 30 19:40:45.087308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:45.087262    4519 generic.go:356] "Generic (PLEG): container finished" podID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerID="736d585cd5aa2d0ac93dc00fc8c5a54be92352c811154edeb9cb4d4ef3e9e31c" exitCode=0
Oct 30 19:40:45.087566 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:45.087416    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m" event={"ID":"3f33a2ae-592e-4750-be1d-827ea1b8d175","Type":"ContainerDied","Data":"736d585cd5aa2d0ac93dc00fc8c5a54be92352c811154edeb9cb4d4ef3e9e31c"}
Oct 30 19:40:45.096065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:45.096030    4519 generic.go:356] "Generic (PLEG): container finished" podID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerID="ef0d7db2cb017afe70b7f212b33bc7b25821bc9137598be1c094b37cfefc8c3d" exitCode=0
Oct 30 19:40:45.096168 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:45.096102    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7" event={"ID":"97b0c2a7-60c7-4431-8d90-e9dd523cf411","Type":"ContainerDied","Data":"ef0d7db2cb017afe70b7f212b33bc7b25821bc9137598be1c094b37cfefc8c3d"}
Oct 30 19:40:46.123607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.123519    4519 generic.go:356] "Generic (PLEG): container finished" podID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerID="79c283ffe201eba20f8cc85a1954a11a5d20fc2fe8dc481a36a25ebbd0258b8a" exitCode=0
Oct 30 19:40:46.124217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.124189    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7" event={"ID":"97b0c2a7-60c7-4431-8d90-e9dd523cf411","Type":"ContainerDied","Data":"79c283ffe201eba20f8cc85a1954a11a5d20fc2fe8dc481a36a25ebbd0258b8a"}
Oct 30 19:40:46.376674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.376585    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:40:46.376839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.376684    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:40:46.376839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.376723    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:40:46.673961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.673124    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:40:46.673961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.673229    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:40:46.673961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:46.673272    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:40:47.140381 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:47.139367    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" event={"ID":"924bd04d-c63b-441d-a17b-fff0e36df522","Type":"ContainerStarted","Data":"e864fa5d7153466d4142734fe14f3f69056fe2b9e66c47364081885c3c751d65"}
Oct 30 19:40:47.158298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:47.157558    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22" event={"ID":"cdd067e2-0384-46e0-b85f-4580b7a107fe","Type":"ContainerStarted","Data":"2ad90af6b70308a1a55e3f05c58bbadfb1142bdeb3460ed5b913675ec6c33bea"}
Oct 30 19:40:47.198209 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:47.198142    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" podStartSLOduration=2.082282589 podStartE2EDuration="11.198121968s" totalImagesPullingTime="9.115839379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:47.197117923 +0000 UTC m=+371.697821645" watchObservedRunningTime="2026-10-30 19:40:47.198121968 +0000 UTC m=+371.698825694"
Oct 30 19:40:47.896093 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:47.896039    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:47.902652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:47.902609    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:47.961134 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:47.961029    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22" podStartSLOduration=2.691198159 podStartE2EDuration="11.961006996s" totalImagesPullingTime="9.269808837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:47.253428791 +0000 UTC m=+371.754132516" watchObservedRunningTime="2026-10-30 19:40:47.961006996 +0000 UTC m=+372.461710723"
Oct 30 19:40:48.026387 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:48.026298    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:48.032905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:48.032869    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:48.166713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:48.166614    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:48.166713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:48.166671    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:48.168916 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:48.168673    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:40:48.170146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:48.170089    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22"
Oct 30 19:40:49.173149 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:49.173109    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerID="630eb9f9b7f58aa2f21a66aefd419d062c70a2dafdf958f9ab5b3ebd9a703c06" exitCode=0
Oct 30 19:40:49.173669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:49.173225    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8" event={"ID":"d5c2aa8a-87b2-406c-81a1-069e66f8f059","Type":"ContainerDied","Data":"630eb9f9b7f58aa2f21a66aefd419d062c70a2dafdf958f9ab5b3ebd9a703c06"}
Oct 30 19:40:49.178963 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:49.178924    4519 generic.go:356] "Generic (PLEG): container finished" podID="702acfe9-e564-4066-840f-a2b4043f8d34" containerID="b4f1f9580d6490905e2704075fa0c80a47e8e6b52f1d16a261857be879ec13cb" exitCode=0
Oct 30 19:40:49.179125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:49.179022    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt" event={"ID":"702acfe9-e564-4066-840f-a2b4043f8d34","Type":"ContainerDied","Data":"b4f1f9580d6490905e2704075fa0c80a47e8e6b52f1d16a261857be879ec13cb"}
Oct 30 19:40:49.183973 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:49.183932    4519 generic.go:356] "Generic (PLEG): container finished" podID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerID="ac480bd1c8df38c16c69dc06309727ec70e8ec4dcb33daa76e28b2147d6b1161" exitCode=0
Oct 30 19:40:49.184621 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:49.184557    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g" event={"ID":"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c","Type":"ContainerDied","Data":"ac480bd1c8df38c16c69dc06309727ec70e8ec4dcb33daa76e28b2147d6b1161"}
Oct 30 19:40:50.190453 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:50.190411    4519 generic.go:356] "Generic (PLEG): container finished" podID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerID="e12804f657d2ebd5cda9e5aad0284ff0bfb605b44bf83d5c08f4cc8d9d746e25" exitCode=0
Oct 30 19:40:50.190908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:50.190797    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz" event={"ID":"9d872bd0-61a4-454c-a4a3-150dac29fe49","Type":"ContainerDied","Data":"e12804f657d2ebd5cda9e5aad0284ff0bfb605b44bf83d5c08f4cc8d9d746e25"}
Oct 30 19:40:51.197907 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:51.197866    4519 generic.go:356] "Generic (PLEG): container finished" podID="1301a737-4486-40b5-99cb-f95577858354" containerID="ac498016eb0dd8b0574c7f99b6404e14a5435ecd1855542e80e0d30e99de0b6d" exitCode=0
Oct 30 19:40:51.198401 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:51.197942    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2" event={"ID":"1301a737-4486-40b5-99cb-f95577858354","Type":"ContainerDied","Data":"ac498016eb0dd8b0574c7f99b6404e14a5435ecd1855542e80e0d30e99de0b6d"}
Oct 30 19:40:51.200187 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:51.200162    4519 generic.go:356] "Generic (PLEG): container finished" podID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerID="43798f3400ec2510f5f5dbbfd57a5a9e42f6f54e12cf74cfc82219dba2671eb9" exitCode=0
Oct 30 19:40:51.200294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:51.200202    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv" event={"ID":"12a0903b-c5ef-4b60-888a-dfcd12f86432","Type":"ContainerDied","Data":"43798f3400ec2510f5f5dbbfd57a5a9e42f6f54e12cf74cfc82219dba2671eb9"}
Oct 30 19:40:52.208110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.208075    4519 generic.go:356] "Generic (PLEG): container finished" podID="1301a737-4486-40b5-99cb-f95577858354" containerID="cb24b575faa50587ab0ba3dd9994bf66cc23f463d42976c5574ebb92b5e8e071" exitCode=0
Oct 30 19:40:52.208676 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.208191    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2" event={"ID":"1301a737-4486-40b5-99cb-f95577858354","Type":"ContainerDied","Data":"cb24b575faa50587ab0ba3dd9994bf66cc23f463d42976c5574ebb92b5e8e071"}
Oct 30 19:40:52.211498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.211432    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerID="6231cc5b0907d9b11639af7d76847e7938a8df0a84bad69d70895d55ddc2a8d3" exitCode=0
Oct 30 19:40:52.211624 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.211512    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8" event={"ID":"d5c2aa8a-87b2-406c-81a1-069e66f8f059","Type":"ContainerDied","Data":"6231cc5b0907d9b11639af7d76847e7938a8df0a84bad69d70895d55ddc2a8d3"}
Oct 30 19:40:52.213831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.213800    4519 generic.go:356] "Generic (PLEG): container finished" podID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerID="1e01f327c5af98118a182375cdee561a2093ea5d88d8c528a9755d10870a3074" exitCode=0
Oct 30 19:40:52.213943 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.213871    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk" event={"ID":"bb69e79d-0af3-45b2-b4b5-5b24d0933e86","Type":"ContainerDied","Data":"1e01f327c5af98118a182375cdee561a2093ea5d88d8c528a9755d10870a3074"}
Oct 30 19:40:52.217098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.217030    4519 generic.go:356] "Generic (PLEG): container finished" podID="702acfe9-e564-4066-840f-a2b4043f8d34" containerID="b7e79d238afe8b57b8bee26d0ea5618cdb96c04188d6908743f7132b17734eb5" exitCode=0
Oct 30 19:40:52.217199 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.217181    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt" event={"ID":"702acfe9-e564-4066-840f-a2b4043f8d34","Type":"ContainerDied","Data":"b7e79d238afe8b57b8bee26d0ea5618cdb96c04188d6908743f7132b17734eb5"}
Oct 30 19:40:52.219652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.219518    4519 generic.go:356] "Generic (PLEG): container finished" podID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerID="10d6a69886fced36db025c91d18e62371b35095e8f19ca5c96c424c4613ccd86" exitCode=0
Oct 30 19:40:52.219652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.219607    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g" event={"ID":"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c","Type":"ContainerDied","Data":"10d6a69886fced36db025c91d18e62371b35095e8f19ca5c96c424c4613ccd86"}
Oct 30 19:40:52.222056 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.222029    4519 generic.go:356] "Generic (PLEG): container finished" podID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerID="b57aca3b1c46c1a5773508a9f130e14778ba9b26dabf2e599dc586f9731c604c" exitCode=0
Oct 30 19:40:52.222143 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:52.222073    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m" event={"ID":"3f33a2ae-592e-4750-be1d-827ea1b8d175","Type":"ContainerDied","Data":"b57aca3b1c46c1a5773508a9f130e14778ba9b26dabf2e599dc586f9731c604c"}
Oct 30 19:40:53.242731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:53.242685    4519 generic.go:356] "Generic (PLEG): container finished" podID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerID="f98088a80fe3b115d12cf2a2f206b5410463779cf55043f00070bf001657e2ff" exitCode=0
Oct 30 19:40:53.243242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:53.242805    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv" event={"ID":"12a0903b-c5ef-4b60-888a-dfcd12f86432","Type":"ContainerDied","Data":"f98088a80fe3b115d12cf2a2f206b5410463779cf55043f00070bf001657e2ff"}
Oct 30 19:40:53.254092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:53.254055    4519 generic.go:356] "Generic (PLEG): container finished" podID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerID="02ea4318eb3aad78a59f09e89c96702c64d5da7b82af8d530bb2b36ff260f434" exitCode=0
Oct 30 19:40:53.254265 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:40:53.254172    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz" event={"ID":"9d872bd0-61a4-454c-a4a3-150dac29fe49","Type":"ContainerDied","Data":"02ea4318eb3aad78a59f09e89c96702c64d5da7b82af8d530bb2b36ff260f434"}
Oct 30 19:41:00.327081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:00.327032    4519 generic.go:356] "Generic (PLEG): container finished" podID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerID="6acfc276bbcbe0f35563e0f5a3c46fb103e9b0fd895853befc5a91f6c07e7027" exitCode=0
Oct 30 19:41:00.327520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:00.327119    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7" event={"ID":"97b0c2a7-60c7-4431-8d90-e9dd523cf411","Type":"ContainerDied","Data":"6acfc276bbcbe0f35563e0f5a3c46fb103e9b0fd895853befc5a91f6c07e7027"}
Oct 30 19:41:01.558725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.558379    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:41:01.734009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.733968    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-util\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-util\") pod \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") "
Oct 30 19:41:01.734195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.734058    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b0c2a7-60c7-4431-8d90-e9dd523cf411-kube-api-access-lbc49\" (UniqueName: \"kubernetes.io/projected/97b0c2a7-60c7-4431-8d90-e9dd523cf411-kube-api-access-lbc49\") pod \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") "
Oct 30 19:41:01.734195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.734104    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-bundle\") pod \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\" (UID: \"97b0c2a7-60c7-4431-8d90-e9dd523cf411\") "
Oct 30 19:41:01.736111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.736076    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-bundle" pod "97b0c2a7-60c7-4431-8d90-e9dd523cf411" (UID: "97b0c2a7-60c7-4431-8d90-e9dd523cf411"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:01.741256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.741211    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-util" pod "97b0c2a7-60c7-4431-8d90-e9dd523cf411" (UID: "97b0c2a7-60c7-4431-8d90-e9dd523cf411"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:01.751895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.750196    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b0c2a7-60c7-4431-8d90-e9dd523cf411-kube-api-access-lbc49" pod "97b0c2a7-60c7-4431-8d90-e9dd523cf411" (UID: "97b0c2a7-60c7-4431-8d90-e9dd523cf411"). InnerVolumeSpecName "kube-api-access-lbc49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:01.835731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.835592    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbc49\" (UniqueName: \"kubernetes.io/projected/97b0c2a7-60c7-4431-8d90-e9dd523cf411-kube-api-access-lbc49\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:01.835731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.835637    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:01.835731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:01.835653    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b0c2a7-60c7-4431-8d90-e9dd523cf411-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:02.266138 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:41:02.265129    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:41:02.360549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:02.355244    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7" event={"ID":"97b0c2a7-60c7-4431-8d90-e9dd523cf411","Type":"ContainerDied","Data":"43bf1933f62b03c5b1d69fa0927f7e4f73c817b8043cf5235968ad55b6486862"}
Oct 30 19:41:02.360549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:02.355306    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43bf1933f62b03c5b1d69fa0927f7e4f73c817b8043cf5235968ad55b6486862"
Oct 30 19:41:02.360549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:02.355399    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrcp7"
Oct 30 19:41:03.366946 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.366875    4519 generic.go:356] "Generic (PLEG): container finished" podID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerID="72da4b34076883a69b2cb520cffa90cc309dd6fcff4287c7d5cc9a684c5ffaf1" exitCode=0
Oct 30 19:41:03.367418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.367392    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m" event={"ID":"3f33a2ae-592e-4750-be1d-827ea1b8d175","Type":"ContainerDied","Data":"72da4b34076883a69b2cb520cffa90cc309dd6fcff4287c7d5cc9a684c5ffaf1"}
Oct 30 19:41:03.370204 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.370167    4519 generic.go:356] "Generic (PLEG): container finished" podID="1301a737-4486-40b5-99cb-f95577858354" containerID="5f0a2dcb3ae8b3fe9821b6d9409314c0c04f95cb0412f2a5ac7377eddbd9be89" exitCode=0
Oct 30 19:41:03.370393 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.370297    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2" event={"ID":"1301a737-4486-40b5-99cb-f95577858354","Type":"ContainerDied","Data":"5f0a2dcb3ae8b3fe9821b6d9409314c0c04f95cb0412f2a5ac7377eddbd9be89"}
Oct 30 19:41:03.373263 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.373047    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerID="f016cf7b9456c6c63bc8058139738f2cc2915f94ec769d7768770dc7b3dc5b14" exitCode=0
Oct 30 19:41:03.373263 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.373161    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8" event={"ID":"d5c2aa8a-87b2-406c-81a1-069e66f8f059","Type":"ContainerDied","Data":"f016cf7b9456c6c63bc8058139738f2cc2915f94ec769d7768770dc7b3dc5b14"}
Oct 30 19:41:03.376284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.376249    4519 generic.go:356] "Generic (PLEG): container finished" podID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerID="867e37144acb7729abdb95b54df3812c5bb96c59ade3ed21c33c24364304f991" exitCode=0
Oct 30 19:41:03.376456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.376340    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk" event={"ID":"bb69e79d-0af3-45b2-b4b5-5b24d0933e86","Type":"ContainerDied","Data":"867e37144acb7729abdb95b54df3812c5bb96c59ade3ed21c33c24364304f991"}
Oct 30 19:41:03.379283 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.379248    4519 generic.go:356] "Generic (PLEG): container finished" podID="702acfe9-e564-4066-840f-a2b4043f8d34" containerID="11c748c57ff1e6056e68453a680fa7542255b2bc48522f90fee120ef75a4b320" exitCode=0
Oct 30 19:41:03.379451 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.379333    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt" event={"ID":"702acfe9-e564-4066-840f-a2b4043f8d34","Type":"ContainerDied","Data":"11c748c57ff1e6056e68453a680fa7542255b2bc48522f90fee120ef75a4b320"}
Oct 30 19:41:03.381570 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.381538    4519 generic.go:356] "Generic (PLEG): container finished" podID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerID="bd482463b854188f9b521097acd9518c57d99f8f97f362430cc2465a70fa7be7" exitCode=0
Oct 30 19:41:03.381733 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:03.381585    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g" event={"ID":"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c","Type":"ContainerDied","Data":"bd482463b854188f9b521097acd9518c57d99f8f97f362430cc2465a70fa7be7"}
Oct 30 19:41:04.389503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.389451    4519 generic.go:356] "Generic (PLEG): container finished" podID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerID="8c3009a10fd04f712d78e4731f9ab8d61d9b9f36e8ccccffa9849fa5b9e80c16" exitCode=0
Oct 30 19:41:04.389969 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.389530    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv" event={"ID":"12a0903b-c5ef-4b60-888a-dfcd12f86432","Type":"ContainerDied","Data":"8c3009a10fd04f712d78e4731f9ab8d61d9b9f36e8ccccffa9849fa5b9e80c16"}
Oct 30 19:41:04.391722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.391684    4519 generic.go:356] "Generic (PLEG): container finished" podID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerID="1139360b20b46a7c185dfdce290c6002c24ca355f545ed7fe134ff450ae9a5de" exitCode=0
Oct 30 19:41:04.391853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.391764    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz" event={"ID":"9d872bd0-61a4-454c-a4a3-150dac29fe49","Type":"ContainerDied","Data":"1139360b20b46a7c185dfdce290c6002c24ca355f545ed7fe134ff450ae9a5de"}
Oct 30 19:41:04.575970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.575927    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:41:04.609693 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.609651    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:41:04.617860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.617835    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:41:04.634502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.629575    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:41:04.634502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.632143    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:41:04.667750 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.667718    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-kube-api-access-d6lkt\" (UniqueName: \"kubernetes.io/projected/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-kube-api-access-d6lkt\") pod \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") "
Oct 30 19:41:04.667921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.667792    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-bundle\") pod \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") "
Oct 30 19:41:04.667921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.667897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-util\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-util\") pod \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\" (UID: \"bb69e79d-0af3-45b2-b4b5-5b24d0933e86\") "
Oct 30 19:41:04.668592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.668359    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-bundle" pod "bb69e79d-0af3-45b2-b4b5-5b24d0933e86" (UID: "bb69e79d-0af3-45b2-b4b5-5b24d0933e86"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.670107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.670079    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-kube-api-access-d6lkt" pod "bb69e79d-0af3-45b2-b4b5-5b24d0933e86" (UID: "bb69e79d-0af3-45b2-b4b5-5b24d0933e86"). InnerVolumeSpecName "kube-api-access-d6lkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:04.671835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.671805    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:41:04.674435 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.674398    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-util" pod "bb69e79d-0af3-45b2-b4b5-5b24d0933e86" (UID: "bb69e79d-0af3-45b2-b4b5-5b24d0933e86"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.769198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.769160    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-kube-api-access-fdf76\" (UniqueName: \"kubernetes.io/projected/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-kube-api-access-fdf76\") pod \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") "
Oct 30 19:41:04.769448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.769213    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1301a737-4486-40b5-99cb-f95577858354-kube-api-access-68vr9\" (UniqueName: \"kubernetes.io/projected/1301a737-4486-40b5-99cb-f95577858354-kube-api-access-68vr9\") pod \"1301a737-4486-40b5-99cb-f95577858354\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") "
Oct 30 19:41:04.769448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.769254    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-bundle\") pod \"3f33a2ae-592e-4750-be1d-827ea1b8d175\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") "
Oct 30 19:41:04.769448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.769406    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-bundle\") pod \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") "
Oct 30 19:41:04.770047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770014    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-bundle" pod "3f33a2ae-592e-4750-be1d-827ea1b8d175" (UID: "3f33a2ae-592e-4750-be1d-827ea1b8d175"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.770163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770103    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702acfe9-e564-4066-840f-a2b4043f8d34-kube-api-access-4bg2w\" (UniqueName: \"kubernetes.io/projected/702acfe9-e564-4066-840f-a2b4043f8d34-kube-api-access-4bg2w\") pod \"702acfe9-e564-4066-840f-a2b4043f8d34\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") "
Oct 30 19:41:04.770163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770138    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-bundle\") pod \"1301a737-4486-40b5-99cb-f95577858354\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") "
Oct 30 19:41:04.770284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770179    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-bundle" pod "d5c2aa8a-87b2-406c-81a1-069e66f8f059" (UID: "d5c2aa8a-87b2-406c-81a1-069e66f8f059"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.770284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770196    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-util\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-util\") pod \"3f33a2ae-592e-4750-be1d-827ea1b8d175\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") "
Oct 30 19:41:04.770284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770248    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c2aa8a-87b2-406c-81a1-069e66f8f059-kube-api-access-j8z6t\" (UniqueName: \"kubernetes.io/projected/d5c2aa8a-87b2-406c-81a1-069e66f8f059-kube-api-access-j8z6t\") pod \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") "
Oct 30 19:41:04.770284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770278    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-util\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-util\") pod \"702acfe9-e564-4066-840f-a2b4043f8d34\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") "
Oct 30 19:41:04.770518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.770309    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-util\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-util\") pod \"1301a737-4486-40b5-99cb-f95577858354\" (UID: \"1301a737-4486-40b5-99cb-f95577858354\") "
Oct 30 19:41:04.771357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.771305    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-bundle" pod "1301a737-4486-40b5-99cb-f95577858354" (UID: "1301a737-4486-40b5-99cb-f95577858354"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.772579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.772553    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-bundle\") pod \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") "
Oct 30 19:41:04.772687 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.772604    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-util\") pod \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\" (UID: \"d5c2aa8a-87b2-406c-81a1-069e66f8f059\") "
Oct 30 19:41:04.772687 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.772633    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-bundle\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-bundle\") pod \"702acfe9-e564-4066-840f-a2b4043f8d34\" (UID: \"702acfe9-e564-4066-840f-a2b4043f8d34\") "
Oct 30 19:41:04.772687 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.772673    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f33a2ae-592e-4750-be1d-827ea1b8d175-kube-api-access-2dkhg\" (UniqueName: \"kubernetes.io/projected/3f33a2ae-592e-4750-be1d-827ea1b8d175-kube-api-access-2dkhg\") pod \"3f33a2ae-592e-4750-be1d-827ea1b8d175\" (UID: \"3f33a2ae-592e-4750-be1d-827ea1b8d175\") "
Oct 30 19:41:04.772842 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.772709    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-util\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-util\") pod \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\" (UID: \"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c\") "
Oct 30 19:41:04.773013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.772997    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lkt\" (UniqueName: \"kubernetes.io/projected/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-kube-api-access-d6lkt\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.773125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.773017    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.773125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.773032    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.773125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.773043    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.773125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.773054    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.773125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.773066    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb69e79d-0af3-45b2-b4b5-5b24d0933e86-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.776732 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.776553    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702acfe9-e564-4066-840f-a2b4043f8d34-kube-api-access-4bg2w" pod "702acfe9-e564-4066-840f-a2b4043f8d34" (UID: "702acfe9-e564-4066-840f-a2b4043f8d34"). InnerVolumeSpecName "kube-api-access-4bg2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:04.777118 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.777093    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1301a737-4486-40b5-99cb-f95577858354-kube-api-access-68vr9" pod "1301a737-4486-40b5-99cb-f95577858354" (UID: "1301a737-4486-40b5-99cb-f95577858354"). InnerVolumeSpecName "kube-api-access-68vr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:04.777809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.777772    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-util" pod "1301a737-4486-40b5-99cb-f95577858354" (UID: "1301a737-4486-40b5-99cb-f95577858354"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.778323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.778296    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-bundle" pod "702acfe9-e564-4066-840f-a2b4043f8d34" (UID: "702acfe9-e564-4066-840f-a2b4043f8d34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.779097 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.779065    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-util" pod "702acfe9-e564-4066-840f-a2b4043f8d34" (UID: "702acfe9-e564-4066-840f-a2b4043f8d34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.779565 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.779504    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-bundle" pod "f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" (UID: "f3c95a41-ea0a-4052-8cee-eb113cc9fb3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.779851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.779800    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-util" pod "3f33a2ae-592e-4750-be1d-827ea1b8d175" (UID: "3f33a2ae-592e-4750-be1d-827ea1b8d175"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.780381 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.780354    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-kube-api-access-fdf76" pod "f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" (UID: "f3c95a41-ea0a-4052-8cee-eb113cc9fb3c"). InnerVolumeSpecName "kube-api-access-fdf76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:04.780670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.780638    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c2aa8a-87b2-406c-81a1-069e66f8f059-kube-api-access-j8z6t" pod "d5c2aa8a-87b2-406c-81a1-069e66f8f059" (UID: "d5c2aa8a-87b2-406c-81a1-069e66f8f059"). InnerVolumeSpecName "kube-api-access-j8z6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:04.781051 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.781030    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f33a2ae-592e-4750-be1d-827ea1b8d175-kube-api-access-2dkhg" pod "3f33a2ae-592e-4750-be1d-827ea1b8d175" (UID: "3f33a2ae-592e-4750-be1d-827ea1b8d175"). InnerVolumeSpecName "kube-api-access-2dkhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:04.781604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.781584    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-util" pod "f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" (UID: "f3c95a41-ea0a-4052-8cee-eb113cc9fb3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.784860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.784819    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-util" pod "d5c2aa8a-87b2-406c-81a1-069e66f8f059" (UID: "d5c2aa8a-87b2-406c-81a1-069e66f8f059"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:04.874083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874045    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874076    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2aa8a-87b2-406c-81a1-069e66f8f059-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874089    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874101    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dkhg\" (UniqueName: \"kubernetes.io/projected/3f33a2ae-592e-4750-be1d-827ea1b8d175-kube-api-access-2dkhg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874114    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874123    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdf76\" (UniqueName: \"kubernetes.io/projected/f3c95a41-ea0a-4052-8cee-eb113cc9fb3c-kube-api-access-fdf76\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874132    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vr9\" (UniqueName: \"kubernetes.io/projected/1301a737-4486-40b5-99cb-f95577858354-kube-api-access-68vr9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874145    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bg2w\" (UniqueName: \"kubernetes.io/projected/702acfe9-e564-4066-840f-a2b4043f8d34-kube-api-access-4bg2w\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874153    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f33a2ae-592e-4750-be1d-827ea1b8d175-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874161    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8z6t\" (UniqueName: \"kubernetes.io/projected/d5c2aa8a-87b2-406c-81a1-069e66f8f059-kube-api-access-j8z6t\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874169    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/702acfe9-e564-4066-840f-a2b4043f8d34-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:04.874324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:04.874180    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1301a737-4486-40b5-99cb-f95577858354-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:05.397788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.397752    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk" event={"ID":"bb69e79d-0af3-45b2-b4b5-5b24d0933e86","Type":"ContainerDied","Data":"1d89809d9d530d079c486178e7c224e9427c4f4fb18f1c0b8ccee494917f498b"}
Oct 30 19:41:05.397788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.397770    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cx9mk"
Oct 30 19:41:05.398337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.397806    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d89809d9d530d079c486178e7c224e9427c4f4fb18f1c0b8ccee494917f498b"
Oct 30 19:41:05.399927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.399904    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt"
Oct 30 19:41:05.400068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.399901    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dxmt" event={"ID":"702acfe9-e564-4066-840f-a2b4043f8d34","Type":"ContainerDied","Data":"a1b9a8ba6de31dbf3b291044a6fcbec2dc36a3cbdb046a53cc578beb7a77581b"}
Oct 30 19:41:05.400068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.400029    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b9a8ba6de31dbf3b291044a6fcbec2dc36a3cbdb046a53cc578beb7a77581b"
Oct 30 19:41:05.402042 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.402020    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g"
Oct 30 19:41:05.402182 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.402033    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fl52g" event={"ID":"f3c95a41-ea0a-4052-8cee-eb113cc9fb3c","Type":"ContainerDied","Data":"7f5e15dd408feceecf2eb126651f3fb6339457494346c4e85578aff0f74cf19f"}
Oct 30 19:41:05.402182 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.402084    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5e15dd408feceecf2eb126651f3fb6339457494346c4e85578aff0f74cf19f"
Oct 30 19:41:05.404343 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.404323    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m"
Oct 30 19:41:05.404343 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.404331    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqv4m" event={"ID":"3f33a2ae-592e-4750-be1d-827ea1b8d175","Type":"ContainerDied","Data":"d6e8dc2d1f8c94cc5a7b7cd0573f6ccaa436506ffa702b496418dbdc34fce486"}
Oct 30 19:41:05.404510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.404372    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e8dc2d1f8c94cc5a7b7cd0573f6ccaa436506ffa702b496418dbdc34fce486"
Oct 30 19:41:05.406723 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.406500    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2"
Oct 30 19:41:05.406723 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.406495    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jwxk2" event={"ID":"1301a737-4486-40b5-99cb-f95577858354","Type":"ContainerDied","Data":"df037babc5508d4fd80b04e185228a4e5444ee7a8d3f67b49aa65e144e7cf52d"}
Oct 30 19:41:05.406723 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.406611    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df037babc5508d4fd80b04e185228a4e5444ee7a8d3f67b49aa65e144e7cf52d"
Oct 30 19:41:05.409485 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.409258    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8"
Oct 30 19:41:05.410976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.410947    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aznzq8" event={"ID":"d5c2aa8a-87b2-406c-81a1-069e66f8f059","Type":"ContainerDied","Data":"985fda29aed7001ff418cdc1b191f6df2515699033e9723c4b44e2e2ae8a8378"}
Oct 30 19:41:05.411072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.410997    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="985fda29aed7001ff418cdc1b191f6df2515699033e9723c4b44e2e2ae8a8378"
Oct 30 19:41:05.591242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.591203    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:41:05.603707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.603675    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:41:05.681809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.681775    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d872bd0-61a4-454c-a4a3-150dac29fe49-kube-api-access-2h2r4\" (UniqueName: \"kubernetes.io/projected/9d872bd0-61a4-454c-a4a3-150dac29fe49-kube-api-access-2h2r4\") pod \"9d872bd0-61a4-454c-a4a3-150dac29fe49\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") "
Oct 30 19:41:05.681986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.681828    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-util\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-util\") pod \"9d872bd0-61a4-454c-a4a3-150dac29fe49\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") "
Oct 30 19:41:05.681986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.681877    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-bundle\") pod \"9d872bd0-61a4-454c-a4a3-150dac29fe49\" (UID: \"9d872bd0-61a4-454c-a4a3-150dac29fe49\") "
Oct 30 19:41:05.682429 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.682394    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-bundle" pod "9d872bd0-61a4-454c-a4a3-150dac29fe49" (UID: "9d872bd0-61a4-454c-a4a3-150dac29fe49"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.684408 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.684379    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d872bd0-61a4-454c-a4a3-150dac29fe49-kube-api-access-2h2r4" pod "9d872bd0-61a4-454c-a4a3-150dac29fe49" (UID: "9d872bd0-61a4-454c-a4a3-150dac29fe49"). InnerVolumeSpecName "kube-api-access-2h2r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:05.687726 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.687697    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-util" pod "9d872bd0-61a4-454c-a4a3-150dac29fe49" (UID: "9d872bd0-61a4-454c-a4a3-150dac29fe49"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.783122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.783088    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-util\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-util\") pod \"12a0903b-c5ef-4b60-888a-dfcd12f86432\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") "
Oct 30 19:41:05.783273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.783199    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-bundle\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-bundle\") pod \"12a0903b-c5ef-4b60-888a-dfcd12f86432\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") "
Oct 30 19:41:05.783273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.783227    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a0903b-c5ef-4b60-888a-dfcd12f86432-kube-api-access-vkm68\" (UniqueName: \"kubernetes.io/projected/12a0903b-c5ef-4b60-888a-dfcd12f86432-kube-api-access-vkm68\") pod \"12a0903b-c5ef-4b60-888a-dfcd12f86432\" (UID: \"12a0903b-c5ef-4b60-888a-dfcd12f86432\") "
Oct 30 19:41:05.783482 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.783432    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2r4\" (UniqueName: \"kubernetes.io/projected/9d872bd0-61a4-454c-a4a3-150dac29fe49-kube-api-access-2h2r4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:05.783482 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.783478    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:05.783670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.783495    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d872bd0-61a4-454c-a4a3-150dac29fe49-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:05.783714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.783666    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-bundle" pod "12a0903b-c5ef-4b60-888a-dfcd12f86432" (UID: "12a0903b-c5ef-4b60-888a-dfcd12f86432"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.785588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.785557    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a0903b-c5ef-4b60-888a-dfcd12f86432-kube-api-access-vkm68" pod "12a0903b-c5ef-4b60-888a-dfcd12f86432" (UID: "12a0903b-c5ef-4b60-888a-dfcd12f86432"). InnerVolumeSpecName "kube-api-access-vkm68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:05.789482 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.789433    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-util" pod "12a0903b-c5ef-4b60-888a-dfcd12f86432" (UID: "12a0903b-c5ef-4b60-888a-dfcd12f86432"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.884898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.884860    4519 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-bundle\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:05.884898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.884892    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkm68\" (UniqueName: \"kubernetes.io/projected/12a0903b-c5ef-4b60-888a-dfcd12f86432-kube-api-access-vkm68\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:05.884898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:05.884904    4519 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12a0903b-c5ef-4b60-888a-dfcd12f86432-util\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:41:06.415175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:06.415136    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv" event={"ID":"12a0903b-c5ef-4b60-888a-dfcd12f86432","Type":"ContainerDied","Data":"7a9acc22d2f91caf9fafa5653db7a9a709ae9bedf64a4ddce01b0b5e8868b326"}
Oct 30 19:41:06.415683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:06.415201    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9acc22d2f91caf9fafa5653db7a9a709ae9bedf64a4ddce01b0b5e8868b326"
Oct 30 19:41:06.415683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:06.415146    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6ftv"
Oct 30 19:41:06.417260 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:06.417230    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz" event={"ID":"9d872bd0-61a4-454c-a4a3-150dac29fe49","Type":"ContainerDied","Data":"9ee4c6d701493317664ac860e19f09ead2f41c50b39a9206f347a1517d98f204"}
Oct 30 19:41:06.417260 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:06.417258    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l49xz"
Oct 30 19:41:06.417429 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:06.417284    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee4c6d701493317664ac860e19f09ead2f41c50b39a9206f347a1517d98f204"
Oct 30 19:41:17.783864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.783824    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"]
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784106    4519 cpu_manager.go:414] "containerMap: removing container" podUID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784117    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784127    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerName="extract"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784132    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerName="extract"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784141    4519 cpu_manager.go:414] "containerMap: removing container" podUID="702acfe9-e564-4066-840f-a2b4043f8d34" containerName="extract"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784146    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702acfe9-e564-4066-840f-a2b4043f8d34" containerName="extract"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784154    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784159    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784166    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784170    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784176    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784181    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784186    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784191    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784196    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784201    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784208    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784212    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784218    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1301a737-4486-40b5-99cb-f95577858354" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784223    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1301a737-4486-40b5-99cb-f95577858354" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784230    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784234    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784240    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerName="extract"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784244    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerName="extract"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784251    4519 cpu_manager.go:414] "containerMap: removing container" podUID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784255    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784261    4519 cpu_manager.go:414] "containerMap: removing container" podUID="702acfe9-e564-4066-840f-a2b4043f8d34" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784265    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702acfe9-e564-4066-840f-a2b4043f8d34" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784272    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784278    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerName="pull"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784284    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1301a737-4486-40b5-99cb-f95577858354" containerName="util"
Oct 30 19:41:17.784274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784288    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1301a737-4486-40b5-99cb-f95577858354" containerName="util"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784294    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784299    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784305    4519 cpu_manager.go:414] "containerMap: removing container" podUID="702acfe9-e564-4066-840f-a2b4043f8d34" containerName="util"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784309    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702acfe9-e564-4066-840f-a2b4043f8d34" containerName="util"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784315    4519 cpu_manager.go:414] "containerMap: removing container" podUID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerName="pull"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784319    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerName="pull"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784328    4519 cpu_manager.go:414] "containerMap: removing container" podUID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784332    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784339    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerName="util"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784343    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerName="util"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784349    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784353    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784361    4519 cpu_manager.go:414] "containerMap: removing container" podUID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784365    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784370    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784375    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784383    4519 cpu_manager.go:414] "containerMap: removing container" podUID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerName="util"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784387    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerName="util"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784395    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerName="pull"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784399    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerName="pull"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784404    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1301a737-4486-40b5-99cb-f95577858354" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784409    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1301a737-4486-40b5-99cb-f95577858354" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784454    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="702acfe9-e564-4066-840f-a2b4043f8d34" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784462    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb69e79d-0af3-45b2-b4b5-5b24d0933e86" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784489    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c2aa8a-87b2-406c-81a1-069e66f8f059" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784496    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a0903b-c5ef-4b60-888a-dfcd12f86432" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784501    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c95a41-ea0a-4052-8cee-eb113cc9fb3c" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784506    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b0c2a7-60c7-4431-8d90-e9dd523cf411" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784512    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f33a2ae-592e-4750-be1d-827ea1b8d175" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784519    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1301a737-4486-40b5-99cb-f95577858354" containerName="extract"
Oct 30 19:41:17.785166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.784525    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d872bd0-61a4-454c-a4a3-150dac29fe49" containerName="extract"
Oct 30 19:41:17.787060 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.787041    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"
Oct 30 19:41:17.806860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.806821    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 19:41:17.807006 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.806943    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:41:17.818239 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.818205    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:41:17.818401 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.818266    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qk4xs\""
Oct 30 19:41:17.821130 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.821097    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"]
Oct 30 19:41:17.879866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.879819    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mxx\" (UniqueName: \"kubernetes.io/projected/777962be-d1c2-4e8d-8aa6-fe0d47c8a514-kube-api-access-79mxx\") pod \"dns-operator-controller-manager-8557cbdb84-v25f9\" (UID: \"777962be-d1c2-4e8d-8aa6-fe0d47c8a514\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"
Oct 30 19:41:17.980531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:17.980462    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mxx\" (UniqueName: \"kubernetes.io/projected/777962be-d1c2-4e8d-8aa6-fe0d47c8a514-kube-api-access-79mxx\") pod \"dns-operator-controller-manager-8557cbdb84-v25f9\" (UID: \"777962be-d1c2-4e8d-8aa6-fe0d47c8a514\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"
Oct 30 19:41:18.004737 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:18.004701    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mxx\" (UniqueName: \"kubernetes.io/projected/777962be-d1c2-4e8d-8aa6-fe0d47c8a514-kube-api-access-79mxx\") pod \"dns-operator-controller-manager-8557cbdb84-v25f9\" (UID: \"777962be-d1c2-4e8d-8aa6-fe0d47c8a514\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"
Oct 30 19:41:18.098408 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:18.098316    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"
Oct 30 19:41:18.422760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:18.422726    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"]
Oct 30 19:41:18.423405 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:18.423378    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777962be_d1c2_4e8d_8aa6_fe0d47c8a514.slice/crio-0cc712ae8db9ab65dede0828a02583bef48b9036359c6d47a638049cfcc68fd5 WatchSource:0}: Error finding container 0cc712ae8db9ab65dede0828a02583bef48b9036359c6d47a638049cfcc68fd5: Status 404 returned error can't find the container with id 0cc712ae8db9ab65dede0828a02583bef48b9036359c6d47a638049cfcc68fd5
Oct 30 19:41:18.469907 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:18.469870    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9" event={"ID":"777962be-d1c2-4e8d-8aa6-fe0d47c8a514","Type":"ContainerStarted","Data":"0cc712ae8db9ab65dede0828a02583bef48b9036359c6d47a638049cfcc68fd5"}
Oct 30 19:41:19.825527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.825433    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"]
Oct 30 19:41:19.829486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.829431    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:19.859245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.859213    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:41:19.859478 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.859440    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 19:41:19.859478 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.859455    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zx96v\""
Oct 30 19:41:19.859618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.859454    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 19:41:19.859618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.859494    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:41:19.859618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.859548    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:41:19.882712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.882677    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"]
Oct 30 19:41:19.899265 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.899231    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7658f\" (UniqueName: \"kubernetes.io/projected/89889f14-3d66-48b1-8389-acb0456502fe-kube-api-access-7658f\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:19.899462 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.899281    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:19.899462 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.899361    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89889f14-3d66-48b1-8389-acb0456502fe-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:19.899565 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.899481    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89889f14-3d66-48b1-8389-acb0456502fe-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:19.899565 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.899532    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:19.899633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:19.899612    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.000391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.000343    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.000589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.000412    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7658f\" (UniqueName: \"kubernetes.io/projected/89889f14-3d66-48b1-8389-acb0456502fe-kube-api-access-7658f\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.000589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.000452    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.000589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.000491    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89889f14-3d66-48b1-8389-acb0456502fe-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.000589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.000529    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89889f14-3d66-48b1-8389-acb0456502fe-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.000589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.000579    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.001177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.001139    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89889f14-3d66-48b1-8389-acb0456502fe-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.003405 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.003375    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89889f14-3d66-48b1-8389-acb0456502fe-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.003618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.003597    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.004121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.004050    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.008680 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.003669    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89889f14-3d66-48b1-8389-acb0456502fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.023028 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.022995    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7658f\" (UniqueName: \"kubernetes.io/projected/89889f14-3d66-48b1-8389-acb0456502fe-kube-api-access-7658f\") pod \"opendatahub-operator-controller-manager-54879d4f6b-lpfww\" (UID: \"89889f14-3d66-48b1-8389-acb0456502fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.148389 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.148304    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:20.573077 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:20.573041    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89889f14_3d66_48b1_8389_acb0456502fe.slice/crio-4d98951b2d1f214c0eff328704c9925c583250498c7fa911b83e6fecd56990f4 WatchSource:0}: Error finding container 4d98951b2d1f214c0eff328704c9925c583250498c7fa911b83e6fecd56990f4: Status 404 returned error can't find the container with id 4d98951b2d1f214c0eff328704c9925c583250498c7fa911b83e6fecd56990f4
Oct 30 19:41:20.582760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:20.582718    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"]
Oct 30 19:41:21.495944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:21.495899    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9" event={"ID":"777962be-d1c2-4e8d-8aa6-fe0d47c8a514","Type":"ContainerStarted","Data":"7a24d4131bf376989546b6487d49bed21191a11770f395c2b8ba89082922b30b"}
Oct 30 19:41:21.496439 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:21.496022    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"
Oct 30 19:41:21.497859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:21.497821    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww" event={"ID":"89889f14-3d66-48b1-8389-acb0456502fe","Type":"ContainerStarted","Data":"4d98951b2d1f214c0eff328704c9925c583250498c7fa911b83e6fecd56990f4"}
Oct 30 19:41:22.795982 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.795922    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9" podStartSLOduration=3.218431546 podStartE2EDuration="5.795906489s" totalImagesPullingTime="2.577474943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:21.540685675 +0000 UTC m=+406.041389399" watchObservedRunningTime="2026-10-30 19:41:22.795906489 +0000 UTC m=+407.296610213"
Oct 30 19:41:22.796871 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.796849    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"]
Oct 30 19:41:22.800839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.800810    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"
Oct 30 19:41:22.827573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.827522    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2t4\" (UniqueName: \"kubernetes.io/projected/f821e59d-1602-4bf8-9167-2ca51e9a8be4-kube-api-access-9c2t4\") pod \"authorino-operator-7b7bb4bc77-8qf89\" (UID: \"f821e59d-1602-4bf8-9167-2ca51e9a8be4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"
Oct 30 19:41:22.830763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.830732    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cplcs\""
Oct 30 19:41:22.833707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.833676    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"]
Oct 30 19:41:22.928121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.928077    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2t4\" (UniqueName: \"kubernetes.io/projected/f821e59d-1602-4bf8-9167-2ca51e9a8be4-kube-api-access-9c2t4\") pod \"authorino-operator-7b7bb4bc77-8qf89\" (UID: \"f821e59d-1602-4bf8-9167-2ca51e9a8be4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"
Oct 30 19:41:22.950385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:22.950353    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2t4\" (UniqueName: \"kubernetes.io/projected/f821e59d-1602-4bf8-9167-2ca51e9a8be4-kube-api-access-9c2t4\") pod \"authorino-operator-7b7bb4bc77-8qf89\" (UID: \"f821e59d-1602-4bf8-9167-2ca51e9a8be4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"
Oct 30 19:41:23.119675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:23.119584    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"
Oct 30 19:41:23.477308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:23.477274    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"]
Oct 30 19:41:23.477619 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:23.477587    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf821e59d_1602_4bf8_9167_2ca51e9a8be4.slice/crio-b16cb155da825ace33f62dbf83d8b4826a1f18c7b47db8fe409fe5c935b46d20 WatchSource:0}: Error finding container b16cb155da825ace33f62dbf83d8b4826a1f18c7b47db8fe409fe5c935b46d20: Status 404 returned error can't find the container with id b16cb155da825ace33f62dbf83d8b4826a1f18c7b47db8fe409fe5c935b46d20
Oct 30 19:41:23.511097 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:23.511059    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89" event={"ID":"f821e59d-1602-4bf8-9167-2ca51e9a8be4","Type":"ContainerStarted","Data":"b16cb155da825ace33f62dbf83d8b4826a1f18c7b47db8fe409fe5c935b46d20"}
Oct 30 19:41:23.512794 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:23.512763    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww" event={"ID":"89889f14-3d66-48b1-8389-acb0456502fe","Type":"ContainerStarted","Data":"874a738606a1341be4ee66849c3d5ba745f485baa67c6b52fdfb7d664dc21335"}
Oct 30 19:41:24.520164 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:24.520115    4519 generic.go:356] "Generic (PLEG): container finished" podID="89889f14-3d66-48b1-8389-acb0456502fe" containerID="874a738606a1341be4ee66849c3d5ba745f485baa67c6b52fdfb7d664dc21335" exitCode=0
Oct 30 19:41:24.520625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:24.520203    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww" event={"ID":"89889f14-3d66-48b1-8389-acb0456502fe","Type":"ContainerDied","Data":"874a738606a1341be4ee66849c3d5ba745f485baa67c6b52fdfb7d664dc21335"}
Oct 30 19:41:25.527235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:25.527191    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89" event={"ID":"f821e59d-1602-4bf8-9167-2ca51e9a8be4","Type":"ContainerStarted","Data":"7e4a050399659d0a7c3fb5de4e14b39a12418fb212e7d5bd0c6115214c0b6552"}
Oct 30 19:41:25.527753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:25.527305    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"
Oct 30 19:41:25.529297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:25.529265    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww" event={"ID":"89889f14-3d66-48b1-8389-acb0456502fe","Type":"ContainerStarted","Data":"861d8286f2e4b7b624f1cadb2d29cda06b20597a90e7b962dfdea9c595b31376"}
Oct 30 19:41:25.529440 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:25.529399    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:25.583574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:25.583507    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89" podStartSLOduration=2.253100335 podStartE2EDuration="3.583491014s" totalImagesPullingTime="1.330390679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:25.58125656 +0000 UTC m=+410.081960283" watchObservedRunningTime="2026-10-30 19:41:25.583491014 +0000 UTC m=+410.084194738"
Oct 30 19:41:25.656595 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:25.656541    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww" podStartSLOduration=3.052258439 podStartE2EDuration="6.656526234s" totalImagesPullingTime="3.094944961s" totalInitContainerRuntime="509.322834ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:25.655789441 +0000 UTC m=+410.156493174" watchObservedRunningTime="2026-10-30 19:41:25.656526234 +0000 UTC m=+410.157229957"
Oct 30 19:41:28.266751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.266715    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"]
Oct 30 19:41:28.270382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.270355    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"
Oct 30 19:41:28.299366 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.299330    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-sgg5n\""
Oct 30 19:41:28.305691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.305134    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"]
Oct 30 19:41:28.377934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.377887    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn27g\" (UniqueName: \"kubernetes.io/projected/12136fec-aa81-4940-adae-bf60070d3589-kube-api-access-qn27g\") pod \"limitador-operator-controller-manager-5778bb4449-4ct7c\" (UID: \"12136fec-aa81-4940-adae-bf60070d3589\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"
Oct 30 19:41:28.479360 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.479319    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn27g\" (UniqueName: \"kubernetes.io/projected/12136fec-aa81-4940-adae-bf60070d3589-kube-api-access-qn27g\") pod \"limitador-operator-controller-manager-5778bb4449-4ct7c\" (UID: \"12136fec-aa81-4940-adae-bf60070d3589\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"
Oct 30 19:41:28.507668 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.507623    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn27g\" (UniqueName: \"kubernetes.io/projected/12136fec-aa81-4940-adae-bf60070d3589-kube-api-access-qn27g\") pod \"limitador-operator-controller-manager-5778bb4449-4ct7c\" (UID: \"12136fec-aa81-4940-adae-bf60070d3589\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"
Oct 30 19:41:28.583379 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.583267    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"
Oct 30 19:41:28.892042 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:28.892002    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"]
Oct 30 19:41:28.893671 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:28.893635    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12136fec_aa81_4940_adae_bf60070d3589.slice/crio-7a3bb27846f9083b0a40b30081663c7ca2a0354b76c432002f49db0ce41c90c4 WatchSource:0}: Error finding container 7a3bb27846f9083b0a40b30081663c7ca2a0354b76c432002f49db0ce41c90c4: Status 404 returned error can't find the container with id 7a3bb27846f9083b0a40b30081663c7ca2a0354b76c432002f49db0ce41c90c4
Oct 30 19:41:29.549133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:29.549094    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c" event={"ID":"12136fec-aa81-4940-adae-bf60070d3589","Type":"ContainerStarted","Data":"7a3bb27846f9083b0a40b30081663c7ca2a0354b76c432002f49db0ce41c90c4"}
Oct 30 19:41:31.559598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:31.559555    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c" event={"ID":"12136fec-aa81-4940-adae-bf60070d3589","Type":"ContainerStarted","Data":"dd2e23758606fa767be824ee24be497d10eed0dc4cbdff6a79134c8317e875de"}
Oct 30 19:41:31.560064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:31.559641    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"
Oct 30 19:41:31.603839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:31.603783    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c" podStartSLOduration=1.8756619140000002 podStartE2EDuration="3.603768571s" totalImagesPullingTime="1.728106657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:31.601929782 +0000 UTC m=+416.102633505" watchObservedRunningTime="2026-10-30 19:41:31.603768571 +0000 UTC m=+416.104472294"
Oct 30 19:41:32.138088 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.138053    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"]
Oct 30 19:41:32.141396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.141373    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.168450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.168404    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.168450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.168432    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.177124 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.177092    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-cdqw2\""
Oct 30 19:41:32.178958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.178934    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"]
Oct 30 19:41:32.212738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.212696    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee7cd19-d41b-4e95-948a-cfd967e48b89-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh\" (UID: \"aee7cd19-d41b-4e95-948a-cfd967e48b89\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.212738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.212735    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvsq\" (UniqueName: \"kubernetes.io/projected/aee7cd19-d41b-4e95-948a-cfd967e48b89-kube-api-access-nmvsq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh\" (UID: \"aee7cd19-d41b-4e95-948a-cfd967e48b89\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.221522 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:41:32.221486    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:41:32.313276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.313237    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee7cd19-d41b-4e95-948a-cfd967e48b89-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh\" (UID: \"aee7cd19-d41b-4e95-948a-cfd967e48b89\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.313276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.313280    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvsq\" (UniqueName: \"kubernetes.io/projected/aee7cd19-d41b-4e95-948a-cfd967e48b89-kube-api-access-nmvsq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh\" (UID: \"aee7cd19-d41b-4e95-948a-cfd967e48b89\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.313685 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.313664    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee7cd19-d41b-4e95-948a-cfd967e48b89-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh\" (UID: \"aee7cd19-d41b-4e95-948a-cfd967e48b89\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.343036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.342991    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvsq\" (UniqueName: \"kubernetes.io/projected/aee7cd19-d41b-4e95-948a-cfd967e48b89-kube-api-access-nmvsq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh\" (UID: \"aee7cd19-d41b-4e95-948a-cfd967e48b89\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.453132 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.453097    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"
Oct 30 19:41:32.508761 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.508677    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v25f9"
Oct 30 19:41:32.776697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:32.776664    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh"]
Oct 30 19:41:32.777352 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:32.777318    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee7cd19_d41b_4e95_948a_cfd967e48b89.slice/crio-8f2bb6b0feb71e1ca05da84cf43e06128b2d6e165103cf9de999ec7869923564 WatchSource:0}: Error finding container 8f2bb6b0feb71e1ca05da84cf43e06128b2d6e165103cf9de999ec7869923564: Status 404 returned error can't find the container with id 8f2bb6b0feb71e1ca05da84cf43e06128b2d6e165103cf9de999ec7869923564
Oct 30 19:41:33.571267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:33.571234    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh" event={"ID":"aee7cd19-d41b-4e95-948a-cfd967e48b89","Type":"ContainerStarted","Data":"8f2bb6b0feb71e1ca05da84cf43e06128b2d6e165103cf9de999ec7869923564"}
Oct 30 19:41:34.577725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:34.577630    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh" event={"ID":"aee7cd19-d41b-4e95-948a-cfd967e48b89","Type":"ContainerStarted","Data":"14a4cb00d58c132489cdd57fc893c5a554cee02d337e99f7f60cc38c56ba0d97"}
Oct 30 19:41:34.646484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:34.646402    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cfxqh" podStartSLOduration=1.149989431 podStartE2EDuration="2.64638008s" totalImagesPullingTime="1.496390649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:34.632858824 +0000 UTC m=+419.133562547" watchObservedRunningTime="2026-10-30 19:41:34.64638008 +0000 UTC m=+419.147083807"
Oct 30 19:41:36.535991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:36.535955    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qf89"
Oct 30 19:41:36.536393 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:36.536041    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww"
Oct 30 19:41:37.738360 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.738320    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"]
Oct 30 19:41:37.743411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.743383    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:37.764158 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.764121    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-465tx\""
Oct 30 19:41:37.777070 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.777036    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"]
Oct 30 19:41:37.861085 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.861043    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvslv\" (UniqueName: \"kubernetes.io/projected/4669b11d-cc77-431d-8f96-80e296a7e193-kube-api-access-mvslv\") pod \"kuadrant-operator-controller-manager-678d87766f-hdfxz\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:37.861269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.861095    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4669b11d-cc77-431d-8f96-80e296a7e193-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hdfxz\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:37.962281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.962234    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4669b11d-cc77-431d-8f96-80e296a7e193-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hdfxz\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:37.962451 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.962359    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvslv\" (UniqueName: \"kubernetes.io/projected/4669b11d-cc77-431d-8f96-80e296a7e193-kube-api-access-mvslv\") pod \"kuadrant-operator-controller-manager-678d87766f-hdfxz\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:37.962809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:37.962790    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4669b11d-cc77-431d-8f96-80e296a7e193-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hdfxz\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:38.001887 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:38.001821    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvslv\" (UniqueName: \"kubernetes.io/projected/4669b11d-cc77-431d-8f96-80e296a7e193-kube-api-access-mvslv\") pod \"kuadrant-operator-controller-manager-678d87766f-hdfxz\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:38.056157 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:38.056116    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:38.404843 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:38.404808    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"]
Oct 30 19:41:38.405116 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:38.405088    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4669b11d_cc77_431d_8f96_80e296a7e193.slice/crio-d1d1b875f2ae7d6594b9f5372aaae7e045abee0c88a9c54013e1f907c0547c2f WatchSource:0}: Error finding container d1d1b875f2ae7d6594b9f5372aaae7e045abee0c88a9c54013e1f907c0547c2f: Status 404 returned error can't find the container with id d1d1b875f2ae7d6594b9f5372aaae7e045abee0c88a9c54013e1f907c0547c2f
Oct 30 19:41:38.599313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:38.599277    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz" event={"ID":"4669b11d-cc77-431d-8f96-80e296a7e193","Type":"ContainerStarted","Data":"d1d1b875f2ae7d6594b9f5372aaae7e045abee0c88a9c54013e1f907c0547c2f"}
Oct 30 19:41:42.567766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:42.567729    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ct7c"
Oct 30 19:41:43.629591 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:43.629553    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz" event={"ID":"4669b11d-cc77-431d-8f96-80e296a7e193","Type":"ContainerStarted","Data":"4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901"}
Oct 30 19:41:43.630035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:43.629629    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:43.701301 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:43.701225    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz" podStartSLOduration=2.194277834 podStartE2EDuration="6.701204342s" totalImagesPullingTime="4.506926508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:43.69975826 +0000 UTC m=+428.200461987" watchObservedRunningTime="2026-10-30 19:41:43.701204342 +0000 UTC m=+428.201908067"
Oct 30 19:41:53.082932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.082886    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xq684"]
Oct 30 19:41:53.091284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.091250    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.115539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.115452    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xq684"]
Oct 30 19:41:53.124868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.124831    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:41:53.124868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.124839    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-526g2\""
Oct 30 19:41:53.184299 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.184265    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"]
Oct 30 19:41:53.188176 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.188147    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.198925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.198397    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbtc\" (UniqueName: \"kubernetes.io/projected/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-kube-api-access-kbbtc\") pod \"limitador-limitador-74dfcfb894-xq684\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.199318 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.199283    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-config-file\") pod \"limitador-limitador-74dfcfb894-xq684\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.215998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.215965    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"]
Oct 30 19:41:53.217778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.217753    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-52qlk\""
Oct 30 19:41:53.230691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.230654    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xq684"]
Oct 30 19:41:53.299770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.299733    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.299971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.299836    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbtc\" (UniqueName: \"kubernetes.io/projected/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-kube-api-access-kbbtc\") pod \"limitador-limitador-74dfcfb894-xq684\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.299971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.299869    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.299971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.299888    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e04538a-4039-4b46-aba2-5fe298998516-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.299971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.299919    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-config-file\") pod \"limitador-limitador-74dfcfb894-xq684\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.300107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.300082    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxsj\" (UniqueName: \"kubernetes.io/projected/6e04538a-4039-4b46-aba2-5fe298998516-kube-api-access-9hxsj\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.300143 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.300118    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.300176 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.300143    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.300290 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.300268    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e04538a-4039-4b46-aba2-5fe298998516-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.300324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.300305    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.300370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.300329    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e04538a-4039-4b46-aba2-5fe298998516-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.300601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.300584    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-config-file\") pod \"limitador-limitador-74dfcfb894-xq684\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.321536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.321500    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbtc\" (UniqueName: \"kubernetes.io/projected/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-kube-api-access-kbbtc\") pod \"limitador-limitador-74dfcfb894-xq684\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.401725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.401636    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.401725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.401676    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e04538a-4039-4b46-aba2-5fe298998516-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.401964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.401808    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.401964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.401882    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.401964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.401912    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e04538a-4039-4b46-aba2-5fe298998516-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.401963    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxsj\" (UniqueName: \"kubernetes.io/projected/6e04538a-4039-4b46-aba2-5fe298998516-kube-api-access-9hxsj\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402007    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402037    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402087    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e04538a-4039-4b46-aba2-5fe298998516-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402149    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402273    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402344    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402454    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e04538a-4039-4b46-aba2-5fe298998516-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.402594 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.402572    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.404385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.404363    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e04538a-4039-4b46-aba2-5fe298998516-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.404741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.404721    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e04538a-4039-4b46-aba2-5fe298998516-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.406607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.406581    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:53.426940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.426911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e04538a-4039-4b46-aba2-5fe298998516-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.429700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.429672    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxsj\" (UniqueName: \"kubernetes.io/projected/6e04538a-4039-4b46-aba2-5fe298998516-kube-api-access-9hxsj\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-p78cx\" (UID: \"6e04538a-4039-4b46-aba2-5fe298998516\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.508240 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.508001    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:53.712108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.712066    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xq684"]
Oct 30 19:41:53.712311 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:53.712230    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e4cc08_b078_479f_85cb_ad9ad0c24fc9.slice/crio-650f1d2a537b3ce1f770072e09b83c998657fb1717a3d1c20c280fa7260365cf WatchSource:0}: Error finding container 650f1d2a537b3ce1f770072e09b83c998657fb1717a3d1c20c280fa7260365cf: Status 404 returned error can't find the container with id 650f1d2a537b3ce1f770072e09b83c998657fb1717a3d1c20c280fa7260365cf
Oct 30 19:41:53.817658 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:53.817622    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e04538a_4039_4b46_aba2_5fe298998516.slice/crio-8de29dd2632698bde2ec86354d9fe578b145c20db865cd2f1ff3a7fc3eaa5c0a WatchSource:0}: Error finding container 8de29dd2632698bde2ec86354d9fe578b145c20db865cd2f1ff3a7fc3eaa5c0a: Status 404 returned error can't find the container with id 8de29dd2632698bde2ec86354d9fe578b145c20db865cd2f1ff3a7fc3eaa5c0a
Oct 30 19:41:53.817856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.817833    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"]
Oct 30 19:41:53.819983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.819944    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:41:53.820098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.820017    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:41:53.820098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:53.820061    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:41:54.055682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.055593    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6z9ts"]
Oct 30 19:41:54.059532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.059499    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6z9ts"
Oct 30 19:41:54.074989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.074955    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhvjj\""
Oct 30 19:41:54.075827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.075790    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6z9ts"]
Oct 30 19:41:54.209978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.209932    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6mb\" (UniqueName: \"kubernetes.io/projected/b0854908-e071-47d1-8968-fe6f1742a969-kube-api-access-mk6mb\") pod \"authorino-57f7f8659f-6z9ts\" (UID: \"b0854908-e071-47d1-8968-fe6f1742a969\") " pod="kuadrant-system/authorino-57f7f8659f-6z9ts"
Oct 30 19:41:54.311159 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.311060    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6mb\" (UniqueName: \"kubernetes.io/projected/b0854908-e071-47d1-8968-fe6f1742a969-kube-api-access-mk6mb\") pod \"authorino-57f7f8659f-6z9ts\" (UID: \"b0854908-e071-47d1-8968-fe6f1742a969\") " pod="kuadrant-system/authorino-57f7f8659f-6z9ts"
Oct 30 19:41:54.331045 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.331007    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6mb\" (UniqueName: \"kubernetes.io/projected/b0854908-e071-47d1-8968-fe6f1742a969-kube-api-access-mk6mb\") pod \"authorino-57f7f8659f-6z9ts\" (UID: \"b0854908-e071-47d1-8968-fe6f1742a969\") " pod="kuadrant-system/authorino-57f7f8659f-6z9ts"
Oct 30 19:41:54.384061 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.384023    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6z9ts"
Oct 30 19:41:54.638025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.637932    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 19:41:54.682690 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.682658    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6z9ts"]
Oct 30 19:41:54.684401 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:41:54.684141    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0854908_e071_47d1_8968_fe6f1742a969.slice/crio-e77b6227b2283eece8ba2f32fe4ac52a9af211863363684c8ad1e31c7931f2a6 WatchSource:0}: Error finding container e77b6227b2283eece8ba2f32fe4ac52a9af211863363684c8ad1e31c7931f2a6: Status 404 returned error can't find the container with id e77b6227b2283eece8ba2f32fe4ac52a9af211863363684c8ad1e31c7931f2a6
Oct 30 19:41:54.692517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.692447    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx" event={"ID":"6e04538a-4039-4b46-aba2-5fe298998516","Type":"ContainerStarted","Data":"a848fbd6bbaa118c11125c32861fd952be61ca1351c269668b3871ed4178efa1"}
Oct 30 19:41:54.692689 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.692545    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx" event={"ID":"6e04538a-4039-4b46-aba2-5fe298998516","Type":"ContainerStarted","Data":"8de29dd2632698bde2ec86354d9fe578b145c20db865cd2f1ff3a7fc3eaa5c0a"}
Oct 30 19:41:54.693890 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.693858    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684" event={"ID":"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9","Type":"ContainerStarted","Data":"650f1d2a537b3ce1f770072e09b83c998657fb1717a3d1c20c280fa7260365cf"}
Oct 30 19:41:54.738247 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:54.738173    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx" podStartSLOduration=1.7381472869999999 podStartE2EDuration="1.738147287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:54.734984447 +0000 UTC m=+439.235688187" watchObservedRunningTime="2026-10-30 19:41:54.738147287 +0000 UTC m=+439.238851013"
Oct 30 19:41:55.508970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.508925    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:55.514444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.514414    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:55.700249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.700201    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6z9ts" event={"ID":"b0854908-e071-47d1-8968-fe6f1742a969","Type":"ContainerStarted","Data":"e77b6227b2283eece8ba2f32fe4ac52a9af211863363684c8ad1e31c7931f2a6"}
Oct 30 19:41:55.701820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.701777    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684" event={"ID":"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9","Type":"ContainerStarted","Data":"39fab4bbb52abb214a496637c8520285757b0ed9078e1a5cdc9fd67549c3f052"}
Oct 30 19:41:55.702035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.702009    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:55.702114 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.702064    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:41:55.702955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.702937    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-p78cx"
Oct 30 19:41:55.743385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:55.743327    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684" podStartSLOduration=1.509035544 podStartE2EDuration="2.743311159s" totalImagesPullingTime="1.234275615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:55.742074393 +0000 UTC m=+440.242778117" watchObservedRunningTime="2026-10-30 19:41:55.743311159 +0000 UTC m=+440.244014886"
Oct 30 19:41:58.845127 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:41:58.845085    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6z9ts"]
Oct 30 19:42:02.220373 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:42:02.220335    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:42:06.709125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:06.709036    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:42:06.757673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:06.757630    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6z9ts" event={"ID":"b0854908-e071-47d1-8968-fe6f1742a969","Type":"ContainerStarted","Data":"0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02"}
Oct 30 19:42:06.757878 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:06.757725    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-6z9ts" podUID="b0854908-e071-47d1-8968-fe6f1742a969" containerName="authorino" containerID="cri-o://0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02" gracePeriod=30
Oct 30 19:42:06.795275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:06.795212    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-6z9ts" podStartSLOduration=1.032568864 podStartE2EDuration="12.795197366s" totalImagesPullingTime="11.762628502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:06.79425664 +0000 UTC m=+451.294960343" watchObservedRunningTime="2026-10-30 19:42:06.795197366 +0000 UTC m=+451.295901089"
Oct 30 19:42:07.013085 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.013052    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6z9ts"
Oct 30 19:42:07.017903 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.017877    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0854908-e071-47d1-8968-fe6f1742a969-kube-api-access-mk6mb\" (UniqueName: \"kubernetes.io/projected/b0854908-e071-47d1-8968-fe6f1742a969-kube-api-access-mk6mb\") pod \"b0854908-e071-47d1-8968-fe6f1742a969\" (UID: \"b0854908-e071-47d1-8968-fe6f1742a969\") "
Oct 30 19:42:07.020140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.020109    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0854908-e071-47d1-8968-fe6f1742a969-kube-api-access-mk6mb" pod "b0854908-e071-47d1-8968-fe6f1742a969" (UID: "b0854908-e071-47d1-8968-fe6f1742a969"). InnerVolumeSpecName "kube-api-access-mk6mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:07.118932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.118895    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk6mb\" (UniqueName: \"kubernetes.io/projected/b0854908-e071-47d1-8968-fe6f1742a969-kube-api-access-mk6mb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:42:07.764366 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.764328    4519 generic.go:356] "Generic (PLEG): container finished" podID="b0854908-e071-47d1-8968-fe6f1742a969" containerID="0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02" exitCode=0
Oct 30 19:42:07.764940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.764374    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6z9ts"
Oct 30 19:42:07.764940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.764419    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6z9ts" event={"ID":"b0854908-e071-47d1-8968-fe6f1742a969","Type":"ContainerDied","Data":"0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02"}
Oct 30 19:42:07.764940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.764492    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6z9ts" event={"ID":"b0854908-e071-47d1-8968-fe6f1742a969","Type":"ContainerDied","Data":"e77b6227b2283eece8ba2f32fe4ac52a9af211863363684c8ad1e31c7931f2a6"}
Oct 30 19:42:07.764940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.764521    4519 scope.go:122] "RemoveContainer" containerID="0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02"
Oct 30 19:42:07.773237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.773216    4519 scope.go:122] "RemoveContainer" containerID="0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02"
Oct 30 19:42:07.773628 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:42:07.773603    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02\": container with ID starting with 0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02 not found: ID does not exist" containerID="0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02"
Oct 30 19:42:07.773694 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.773640    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02"} err="failed to get container status \"0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02\": rpc error: code = NotFound desc = could not find container \"0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02\": container with ID starting with 0df9e55a4b3bde680a414744135fe12d784fbb77ea8f69d397c152302b16cb02 not found: ID does not exist"
Oct 30 19:42:07.816938 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.816900    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6z9ts"]
Oct 30 19:42:07.822096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:07.822065    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6z9ts"]
Oct 30 19:42:08.366496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:08.366444    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xq684"]
Oct 30 19:42:08.366727 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:08.366688    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684" podUID="f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" containerName="limitador" containerID="cri-o://39fab4bbb52abb214a496637c8520285757b0ed9078e1a5cdc9fd67549c3f052" gracePeriod=30
Oct 30 19:42:08.771301 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:08.771267    4519 generic.go:356] "Generic (PLEG): container finished" podID="f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" containerID="39fab4bbb52abb214a496637c8520285757b0ed9078e1a5cdc9fd67549c3f052" exitCode=0
Oct 30 19:42:08.771814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:08.771341    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684" event={"ID":"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9","Type":"ContainerDied","Data":"39fab4bbb52abb214a496637c8520285757b0ed9078e1a5cdc9fd67549c3f052"}
Oct 30 19:42:09.321674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.321643    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:42:09.336505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.336456    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-kube-api-access-kbbtc\" (UniqueName: \"kubernetes.io/projected/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-kube-api-access-kbbtc\") pod \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") "
Oct 30 19:42:09.338715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.338683    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-kube-api-access-kbbtc" pod "f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" (UID: "f3e4cc08-b078-479f-85cb-ad9ad0c24fc9"). InnerVolumeSpecName "kube-api-access-kbbtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:09.436996 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.436907    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-config-file\" (UniqueName: \"kubernetes.io/configmap/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-config-file\") pod \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\" (UID: \"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9\") "
Oct 30 19:42:09.437155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.437069    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbtc\" (UniqueName: \"kubernetes.io/projected/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-kube-api-access-kbbtc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:42:09.437283 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.437255    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-config-file" pod "f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" (UID: "f3e4cc08-b078-479f-85cb-ad9ad0c24fc9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:09.537636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.537590    4519 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9-config-file\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:42:09.610660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.610624    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0854908-e071-47d1-8968-fe6f1742a969" path="/var/lib/kubelet/pods/b0854908-e071-47d1-8968-fe6f1742a969/volumes"
Oct 30 19:42:09.779349 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.779253    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684" event={"ID":"f3e4cc08-b078-479f-85cb-ad9ad0c24fc9","Type":"ContainerDied","Data":"650f1d2a537b3ce1f770072e09b83c998657fb1717a3d1c20c280fa7260365cf"}
Oct 30 19:42:09.779349 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.779323    4519 scope.go:122] "RemoveContainer" containerID="39fab4bbb52abb214a496637c8520285757b0ed9078e1a5cdc9fd67549c3f052"
Oct 30 19:42:09.779997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.779271    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xq684"
Oct 30 19:42:09.834484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.834432    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xq684"]
Oct 30 19:42:09.844755 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:09.844710    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xq684"]
Oct 30 19:42:11.609989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:11.609955    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" path="/var/lib/kubelet/pods/f3e4cc08-b078-479f-85cb-ad9ad0c24fc9/volumes"
Oct 30 19:42:15.662350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.662315    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"]
Oct 30 19:42:15.662753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.662678    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b0854908-e071-47d1-8968-fe6f1742a969" containerName="authorino"
Oct 30 19:42:15.662753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.662689    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0854908-e071-47d1-8968-fe6f1742a969" containerName="authorino"
Oct 30 19:42:15.662753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.662706    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" containerName="limitador"
Oct 30 19:42:15.662753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.662711    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" containerName="limitador"
Oct 30 19:42:15.662888 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.662770    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e4cc08-b078-479f-85cb-ad9ad0c24fc9" containerName="limitador"
Oct 30 19:42:15.662888 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.662782    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0854908-e071-47d1-8968-fe6f1742a969" containerName="authorino"
Oct 30 19:42:15.665581 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.665460    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:15.679144 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.679112    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/88f79d2d-2552-4139-8fef-c2af42193e46-data\") pod \"postgres-7cdbf6dc44-m7fmb\" (UID: \"88f79d2d-2552-4139-8fef-c2af42193e46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:15.679322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.679191    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfg6\" (UniqueName: \"kubernetes.io/projected/88f79d2d-2552-4139-8fef-c2af42193e46-kube-api-access-nnfg6\") pod \"postgres-7cdbf6dc44-m7fmb\" (UID: \"88f79d2d-2552-4139-8fef-c2af42193e46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:15.688683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.688637    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:15.688683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.688643    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:15.700805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.700770    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ks5hm\""
Oct 30 19:42:15.700805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.700770    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 19:42:15.702864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.702832    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"]
Oct 30 19:42:15.779800 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.779757    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/88f79d2d-2552-4139-8fef-c2af42193e46-data\") pod \"postgres-7cdbf6dc44-m7fmb\" (UID: \"88f79d2d-2552-4139-8fef-c2af42193e46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:15.779985 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.779842    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfg6\" (UniqueName: \"kubernetes.io/projected/88f79d2d-2552-4139-8fef-c2af42193e46-kube-api-access-nnfg6\") pod \"postgres-7cdbf6dc44-m7fmb\" (UID: \"88f79d2d-2552-4139-8fef-c2af42193e46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:15.780179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.780156    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/88f79d2d-2552-4139-8fef-c2af42193e46-data\") pod \"postgres-7cdbf6dc44-m7fmb\" (UID: \"88f79d2d-2552-4139-8fef-c2af42193e46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:15.800585 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.800553    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfg6\" (UniqueName: \"kubernetes.io/projected/88f79d2d-2552-4139-8fef-c2af42193e46-kube-api-access-nnfg6\") pod \"postgres-7cdbf6dc44-m7fmb\" (UID: \"88f79d2d-2552-4139-8fef-c2af42193e46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:15.978227 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:15.978185    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:16.283065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:16.283036    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"]
Oct 30 19:42:16.284623 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:42:16.284593    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f79d2d_2552_4139_8fef_c2af42193e46.slice/crio-be42e1f36f64e02e1ac3591a0da97ac844ec6334cf9dadc7a308e964dd7bb11b WatchSource:0}: Error finding container be42e1f36f64e02e1ac3591a0da97ac844ec6334cf9dadc7a308e964dd7bb11b: Status 404 returned error can't find the container with id be42e1f36f64e02e1ac3591a0da97ac844ec6334cf9dadc7a308e964dd7bb11b
Oct 30 19:42:16.816515 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:16.816456    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb" event={"ID":"88f79d2d-2552-4139-8fef-c2af42193e46","Type":"ContainerStarted","Data":"be42e1f36f64e02e1ac3591a0da97ac844ec6334cf9dadc7a308e964dd7bb11b"}
Oct 30 19:42:21.847501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:21.847426    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb" event={"ID":"88f79d2d-2552-4139-8fef-c2af42193e46","Type":"ContainerStarted","Data":"9341299e91dd1c82a090562a15f4ba2bb705dacdff30208622c08151d5fe00dd"}
Oct 30 19:42:21.848158 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:21.847518    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:21.889011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:21.888951    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb" podStartSLOduration=2.205616469 podStartE2EDuration="6.88893572s" totalImagesPullingTime="4.683319251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:21.887849014 +0000 UTC m=+466.388552741" watchObservedRunningTime="2026-10-30 19:42:21.88893572 +0000 UTC m=+466.389639443"
Oct 30 19:42:27.884220 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:27.884184    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m7fmb"
Oct 30 19:42:32.221130 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:42:32.221087    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:42:35.105148 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.105100    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f5dn9"]
Oct 30 19:42:35.111371 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.111344    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.142616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.142462    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 19:42:35.142616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.142491    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 19:42:35.142616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.142521    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nw5kj\""
Oct 30 19:42:35.142616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.142522    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:42:35.142616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.142572    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:42:35.142616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.142584    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:42:35.145599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.145567    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f5dn9"]
Oct 30 19:42:35.260231 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.260189    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-webhook-cert\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.260231 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.260229    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-metrics-certs\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.260454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.260394    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwwbv\" (UniqueName: \"kubernetes.io/projected/daa01d27-70d0-433f-b54e-1d4e908f01b9-kube-api-access-wwwbv\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.361404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.361312    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-webhook-cert\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.361404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.361351    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-metrics-certs\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.361404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.361393    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwwbv\" (UniqueName: \"kubernetes.io/projected/daa01d27-70d0-433f-b54e-1d4e908f01b9-kube-api-access-wwwbv\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.364196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.364159    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-metrics-certs\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.364331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.364159    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-webhook-cert\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.382620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.382587    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwwbv\" (UniqueName: \"kubernetes.io/projected/daa01d27-70d0-433f-b54e-1d4e908f01b9-kube-api-access-wwwbv\") pod \"maas-controller-579cf7c6d-f5dn9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") " pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.425789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.425749    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:35.448602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.446812    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f86dcf78c-xbjnv"]
Oct 30 19:42:35.454361 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.454330    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.472042 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.472005    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-xbjnv"]
Oct 30 19:42:35.563195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.563155    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96w6v\" (UniqueName: \"kubernetes.io/projected/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-kube-api-access-96w6v\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.563195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.563205    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-webhook-cert\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.563495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.563311    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-metrics-certs\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.663819 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.663783    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96w6v\" (UniqueName: \"kubernetes.io/projected/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-kube-api-access-96w6v\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.663980 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.663833    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-webhook-cert\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.663980 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.663863    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-metrics-certs\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.666526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.666499    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-webhook-cert\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.666658 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.666508    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-metrics-certs\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.684654 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.684618    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96w6v\" (UniqueName: \"kubernetes.io/projected/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-kube-api-access-96w6v\") pod \"maas-controller-f86dcf78c-xbjnv\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") " pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.737971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.737942    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f5dn9"]
Oct 30 19:42:35.738865 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:42:35.738843    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa01d27_70d0_433f_b54e_1d4e908f01b9.slice/crio-e533bfa0c3e78c2266be58c49e8c1a46053ed2e2bdea4bd4ae281a6e3a20b71c WatchSource:0}: Error finding container e533bfa0c3e78c2266be58c49e8c1a46053ed2e2bdea4bd4ae281a6e3a20b71c: Status 404 returned error can't find the container with id e533bfa0c3e78c2266be58c49e8c1a46053ed2e2bdea4bd4ae281a6e3a20b71c
Oct 30 19:42:35.771524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.771487    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:35.917579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:35.917543    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f5dn9" event={"ID":"daa01d27-70d0-433f-b54e-1d4e908f01b9","Type":"ContainerStarted","Data":"e533bfa0c3e78c2266be58c49e8c1a46053ed2e2bdea4bd4ae281a6e3a20b71c"}
Oct 30 19:42:36.078804 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:36.078775    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-xbjnv"]
Oct 30 19:42:36.079391 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:42:36.079369    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9324bb_be21_4f9f_a5cd_a983a2d585aa.slice/crio-7035bcb64c049a38628b5449a5abadb7c2f96e6cd6f8b044a5cb6003e7305d61 WatchSource:0}: Error finding container 7035bcb64c049a38628b5449a5abadb7c2f96e6cd6f8b044a5cb6003e7305d61: Status 404 returned error can't find the container with id 7035bcb64c049a38628b5449a5abadb7c2f96e6cd6f8b044a5cb6003e7305d61
Oct 30 19:42:36.928805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:36.928709    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" event={"ID":"ae9324bb-be21-4f9f-a5cd-a983a2d585aa","Type":"ContainerStarted","Data":"7035bcb64c049a38628b5449a5abadb7c2f96e6cd6f8b044a5cb6003e7305d61"}
Oct 30 19:42:38.704767 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:38.704729    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:42:38.943605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:38.943507    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f5dn9" event={"ID":"daa01d27-70d0-433f-b54e-1d4e908f01b9","Type":"ContainerStarted","Data":"1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf"}
Oct 30 19:42:38.943775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:38.943629    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:38.945091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:38.945068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" event={"ID":"ae9324bb-be21-4f9f-a5cd-a983a2d585aa","Type":"ContainerStarted","Data":"f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69"}
Oct 30 19:42:38.945221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:38.945205    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:38.996237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:38.996181    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-f5dn9" podStartSLOduration=1.046121263 podStartE2EDuration="3.996163723s" totalImagesPullingTime="2.95004246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:38.99421523 +0000 UTC m=+483.494918954" watchObservedRunningTime="2026-10-30 19:42:38.996163723 +0000 UTC m=+483.496867447"
Oct 30 19:42:39.046094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:39.046028    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" podStartSLOduration=1.29691431 podStartE2EDuration="4.04601344s" totalImagesPullingTime="2.74909913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:39.045741235 +0000 UTC m=+483.546444959" watchObservedRunningTime="2026-10-30 19:42:39.04601344 +0000 UTC m=+483.546717165"
Oct 30 19:42:40.529342 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.529301    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"]
Oct 30 19:42:40.532881 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.532858    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.565316 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.565283    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 19:42:40.565569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.565294    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 19:42:40.565569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.565371    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-jcm9j\""
Oct 30 19:42:40.565758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.565376    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 19:42:40.568028 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.567999    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"]
Oct 30 19:42:40.616153 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.616109    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czbsh\" (UniqueName: \"kubernetes.io/projected/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-kube-api-access-czbsh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.616331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.616242    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.616331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.616288    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.616331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.616312    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.717197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.717155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czbsh\" (UniqueName: \"kubernetes.io/projected/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-kube-api-access-czbsh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.717417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.717253    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.717417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.717295    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.717417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.717327    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.718209 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.718171    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.718428 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.718409    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.720154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.720125    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.738200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.738168    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czbsh\" (UniqueName: \"kubernetes.io/projected/b2e9940f-4bb3-44c9-9dad-d71f47e4d627-kube-api-access-czbsh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5qgtc\" (UID: \"b2e9940f-4bb3-44c9-9dad-d71f47e4d627\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:40.848370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:40.848279    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:41.151182 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:41.151155    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"]
Oct 30 19:42:41.152163 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:42:41.152136    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e9940f_4bb3_44c9_9dad_d71f47e4d627.slice/crio-ac4423beaed610bde5eb61c9285ba8e8917c817eeac290d78fa28649d3c2a1ad WatchSource:0}: Error finding container ac4423beaed610bde5eb61c9285ba8e8917c817eeac290d78fa28649d3c2a1ad: Status 404 returned error can't find the container with id ac4423beaed610bde5eb61c9285ba8e8917c817eeac290d78fa28649d3c2a1ad
Oct 30 19:42:41.962090 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:41.962051    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc" event={"ID":"b2e9940f-4bb3-44c9-9dad-d71f47e4d627","Type":"ContainerStarted","Data":"ac4423beaed610bde5eb61c9285ba8e8917c817eeac290d78fa28649d3c2a1ad"}
Oct 30 19:42:49.952877 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:49.952795    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:42:49.952877 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:49.952872    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:50.123504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.123451    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f5dn9"]
Oct 30 19:42:50.123758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.123728    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-f5dn9" podUID="daa01d27-70d0-433f-b54e-1d4e908f01b9" containerName="manager" containerID="cri-o://1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf" gracePeriod=30
Oct 30 19:42:50.385077 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.385043    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:50.510530 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.510424    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-metrics-certs\") pod \"daa01d27-70d0-433f-b54e-1d4e908f01b9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") "
Oct 30 19:42:50.510700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.510540    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-webhook-cert\") pod \"daa01d27-70d0-433f-b54e-1d4e908f01b9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") "
Oct 30 19:42:50.510700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.510604    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa01d27-70d0-433f-b54e-1d4e908f01b9-kube-api-access-wwwbv\" (UniqueName: \"kubernetes.io/projected/daa01d27-70d0-433f-b54e-1d4e908f01b9-kube-api-access-wwwbv\") pod \"daa01d27-70d0-433f-b54e-1d4e908f01b9\" (UID: \"daa01d27-70d0-433f-b54e-1d4e908f01b9\") "
Oct 30 19:42:50.513179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.513137    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-webhook-cert" pod "daa01d27-70d0-433f-b54e-1d4e908f01b9" (UID: "daa01d27-70d0-433f-b54e-1d4e908f01b9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:50.513320 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.513144    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa01d27-70d0-433f-b54e-1d4e908f01b9-kube-api-access-wwwbv" pod "daa01d27-70d0-433f-b54e-1d4e908f01b9" (UID: "daa01d27-70d0-433f-b54e-1d4e908f01b9"). InnerVolumeSpecName "kube-api-access-wwwbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:50.515880 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.515831    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-metrics-certs" pod "daa01d27-70d0-433f-b54e-1d4e908f01b9" (UID: "daa01d27-70d0-433f-b54e-1d4e908f01b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:50.611614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.611576    4519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-metrics-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:42:50.611614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.611611    4519 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa01d27-70d0-433f-b54e-1d4e908f01b9-webhook-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:42:50.611858 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:50.611627    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwwbv\" (UniqueName: \"kubernetes.io/projected/daa01d27-70d0-433f-b54e-1d4e908f01b9-kube-api-access-wwwbv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:42:51.018808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.018767    4519 generic.go:356] "Generic (PLEG): container finished" podID="daa01d27-70d0-433f-b54e-1d4e908f01b9" containerID="1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf" exitCode=0
Oct 30 19:42:51.019281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.018819    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f5dn9"
Oct 30 19:42:51.019281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.018828    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f5dn9" event={"ID":"daa01d27-70d0-433f-b54e-1d4e908f01b9","Type":"ContainerDied","Data":"1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf"}
Oct 30 19:42:51.019281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.018881    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f5dn9" event={"ID":"daa01d27-70d0-433f-b54e-1d4e908f01b9","Type":"ContainerDied","Data":"e533bfa0c3e78c2266be58c49e8c1a46053ed2e2bdea4bd4ae281a6e3a20b71c"}
Oct 30 19:42:51.019281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.018913    4519 scope.go:122] "RemoveContainer" containerID="1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf"
Oct 30 19:42:51.035198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.035061    4519 scope.go:122] "RemoveContainer" containerID="1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf"
Oct 30 19:42:51.035549 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:42:51.035506    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf\": container with ID starting with 1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf not found: ID does not exist" containerID="1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf"
Oct 30 19:42:51.035684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.035564    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf"} err="failed to get container status \"1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf\": rpc error: code = NotFound desc = could not find container \"1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf\": container with ID starting with 1d83a97b75d22273321eb8b972fd3cf464818ecb125a0a2c5a99b6b4805b8daf not found: ID does not exist"
Oct 30 19:42:51.113760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.113719    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f5dn9"]
Oct 30 19:42:51.125841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.125810    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f5dn9"]
Oct 30 19:42:51.612683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:51.612644    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa01d27-70d0-433f-b54e-1d4e908f01b9" path="/var/lib/kubelet/pods/daa01d27-70d0-433f-b54e-1d4e908f01b9/volumes"
Oct 30 19:42:53.033014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:53.032969    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc" event={"ID":"b2e9940f-4bb3-44c9-9dad-d71f47e4d627","Type":"ContainerStarted","Data":"b38fa2406fa74aeefee0122f92dc3b16cf2cc96c644597033c1628d60cab742d"}
Oct 30 19:42:53.033495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:53.033034    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:42:58.799699 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.799603    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc" podStartSLOduration=7.337950552 podStartE2EDuration="18.799585255s" totalImagesPullingTime="11.461634703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:53.100979314 +0000 UTC m=+497.601683039" watchObservedRunningTime="2026-10-30 19:42:58.799585255 +0000 UTC m=+503.300288978"
Oct 30 19:42:58.800951 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.800921    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vmkgb"]
Oct 30 19:42:58.801294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.801281    4519 cpu_manager.go:414] "containerMap: removing container" podUID="daa01d27-70d0-433f-b54e-1d4e908f01b9" containerName="manager"
Oct 30 19:42:58.801352 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.801296    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa01d27-70d0-433f-b54e-1d4e908f01b9" containerName="manager"
Oct 30 19:42:58.801389 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.801371    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa01d27-70d0-433f-b54e-1d4e908f01b9" containerName="manager"
Oct 30 19:42:58.811814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.811777    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:58.838558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.835867    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 19:42:58.838558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.835988    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7fxfz\""
Oct 30 19:42:58.838558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.836117    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vmkgb"]
Oct 30 19:42:58.864886 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.864843    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"]
Oct 30 19:42:58.868491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.868452    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:58.901531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.901496    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"]
Oct 30 19:42:58.930297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.930261    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 19:42:58.930510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.930347    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-26dcq\""
Oct 30 19:42:58.956200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.956159    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-xxgh4"]
Oct 30 19:42:58.959922 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.959900    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:58.983046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.983010    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xxgh4"]
Oct 30 19:42:58.991917 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.991866    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/951ab11b-0442-4de2-93d8-5d4e09759ac7-cert\") pod \"kserve-controller-manager-776cc66d47-vmkgb\" (UID: \"951ab11b-0442-4de2-93d8-5d4e09759ac7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:58.992090 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.991922    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ss6v\" (UniqueName: \"kubernetes.io/projected/39adede1-163e-45df-b381-ab419e3a433a-kube-api-access-4ss6v\") pod \"llmisvc-controller-manager-869d5dcc5d-m4hnn\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:58.992090 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.992078    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948c9\" (UniqueName: \"kubernetes.io/projected/951ab11b-0442-4de2-93d8-5d4e09759ac7-kube-api-access-948c9\") pod \"kserve-controller-manager-776cc66d47-vmkgb\" (UID: \"951ab11b-0442-4de2-93d8-5d4e09759ac7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:58.992161 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:58.992105    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39adede1-163e-45df-b381-ab419e3a433a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-m4hnn\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:59.030526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.030485    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6wgxw\""
Oct 30 19:42:59.030526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.030500    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 19:42:59.030751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.030505    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 19:42:59.093576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.093442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/951ab11b-0442-4de2-93d8-5d4e09759ac7-cert\") pod \"kserve-controller-manager-776cc66d47-vmkgb\" (UID: \"951ab11b-0442-4de2-93d8-5d4e09759ac7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:59.093576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.093525    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ss6v\" (UniqueName: \"kubernetes.io/projected/39adede1-163e-45df-b381-ab419e3a433a-kube-api-access-4ss6v\") pod \"llmisvc-controller-manager-869d5dcc5d-m4hnn\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:59.093576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.093562    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-metrics-certs\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.093873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.093605    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-cert\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.093873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.093726    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948c9\" (UniqueName: \"kubernetes.io/projected/951ab11b-0442-4de2-93d8-5d4e09759ac7-kube-api-access-948c9\") pod \"kserve-controller-manager-776cc66d47-vmkgb\" (UID: \"951ab11b-0442-4de2-93d8-5d4e09759ac7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:59.093873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.093745    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39adede1-163e-45df-b381-ab419e3a433a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-m4hnn\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:59.093873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.093763    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfsrz\" (UniqueName: \"kubernetes.io/projected/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-kube-api-access-lfsrz\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.096372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.096336    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39adede1-163e-45df-b381-ab419e3a433a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-m4hnn\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:59.096561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.096372    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/951ab11b-0442-4de2-93d8-5d4e09759ac7-cert\") pod \"kserve-controller-manager-776cc66d47-vmkgb\" (UID: \"951ab11b-0442-4de2-93d8-5d4e09759ac7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:59.121194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.121159    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ss6v\" (UniqueName: \"kubernetes.io/projected/39adede1-163e-45df-b381-ab419e3a433a-kube-api-access-4ss6v\") pod \"llmisvc-controller-manager-869d5dcc5d-m4hnn\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:59.122826 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.122798    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-948c9\" (UniqueName: \"kubernetes.io/projected/951ab11b-0442-4de2-93d8-5d4e09759ac7-kube-api-access-948c9\") pod \"kserve-controller-manager-776cc66d47-vmkgb\" (UID: \"951ab11b-0442-4de2-93d8-5d4e09759ac7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:59.125625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.125597    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:42:59.182065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.182025    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:42:59.195933 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.194518    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-metrics-certs\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.195933 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.194594    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-cert\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.195933 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.194671    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfsrz\" (UniqueName: \"kubernetes.io/projected/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-kube-api-access-lfsrz\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.199245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.199126    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-metrics-certs\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.199897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.199851    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-cert\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.219668 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.219619    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfsrz\" (UniqueName: \"kubernetes.io/projected/67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb-kube-api-access-lfsrz\") pod \"odh-model-controller-769875db56-xxgh4\" (UID: \"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb\") " pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.273929 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.273889    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:42:59.429973 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.429939    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vmkgb"]
Oct 30 19:42:59.430094 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:42:59.430047    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951ab11b_0442_4de2_93d8_5d4e09759ac7.slice/crio-a7e914a15dfb6f8145a8686a14844dada32ba27accc4eb4f97cb688eb540744e WatchSource:0}: Error finding container a7e914a15dfb6f8145a8686a14844dada32ba27accc4eb4f97cb688eb540744e: Status 404 returned error can't find the container with id a7e914a15dfb6f8145a8686a14844dada32ba27accc4eb4f97cb688eb540744e
Oct 30 19:42:59.516030 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.516001    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"]
Oct 30 19:42:59.517993 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:42:59.517962    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod39adede1_163e_45df_b381_ab419e3a433a.slice/crio-bc447726fd3c69f41ef215d8383ea5e91fad7d0ce843736c82f4eeb6377ddf8a WatchSource:0}: Error finding container bc447726fd3c69f41ef215d8383ea5e91fad7d0ce843736c82f4eeb6377ddf8a: Status 404 returned error can't find the container with id bc447726fd3c69f41ef215d8383ea5e91fad7d0ce843736c82f4eeb6377ddf8a
Oct 30 19:42:59.583690 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:42:59.583651    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xxgh4"]
Oct 30 19:42:59.585199 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:42:59.585169    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67469f7a_1c81_4e4e_9ab6_ddbd86b3d6fb.slice/crio-300e33606ba586acf30ba069c5654cb5c52d06f6e129458fdeacdd6961079299 WatchSource:0}: Error finding container 300e33606ba586acf30ba069c5654cb5c52d06f6e129458fdeacdd6961079299: Status 404 returned error can't find the container with id 300e33606ba586acf30ba069c5654cb5c52d06f6e129458fdeacdd6961079299
Oct 30 19:43:00.072292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:00.072211    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xxgh4" event={"ID":"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb","Type":"ContainerStarted","Data":"300e33606ba586acf30ba069c5654cb5c52d06f6e129458fdeacdd6961079299"}
Oct 30 19:43:00.074518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:00.074405    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn" event={"ID":"39adede1-163e-45df-b381-ab419e3a433a","Type":"ContainerStarted","Data":"bc447726fd3c69f41ef215d8383ea5e91fad7d0ce843736c82f4eeb6377ddf8a"}
Oct 30 19:43:00.076611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:00.076529    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb" event={"ID":"951ab11b-0442-4de2-93d8-5d4e09759ac7","Type":"ContainerStarted","Data":"a7e914a15dfb6f8145a8686a14844dada32ba27accc4eb4f97cb688eb540744e"}
Oct 30 19:43:02.232826 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:43:02.232775    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:43:04.043349 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.043310    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5qgtc"
Oct 30 19:43:04.110481 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.110416    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb" event={"ID":"951ab11b-0442-4de2-93d8-5d4e09759ac7","Type":"ContainerStarted","Data":"bb6e22f72a6e8e309420d2c38b78836e92e3bb67bfce38358c4889a767c957e5"}
Oct 30 19:43:04.110748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.110522    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:43:04.112838 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.112795    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xxgh4" event={"ID":"67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb","Type":"ContainerStarted","Data":"8ca52005cf5271ad5d7b82454213ae88f55b92cd65e1420f97735df1dca88cd2"}
Oct 30 19:43:04.112993 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.112937    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:43:04.115035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.114991    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn" event={"ID":"39adede1-163e-45df-b381-ab419e3a433a","Type":"ContainerStarted","Data":"db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe"}
Oct 30 19:43:04.115170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.115067    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:43:04.152660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.152599    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb" podStartSLOduration=2.316571158 podStartE2EDuration="6.152584635s" totalImagesPullingTime="3.836013477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:04.151866286 +0000 UTC m=+508.652570012" watchObservedRunningTime="2026-10-30 19:43:04.152584635 +0000 UTC m=+508.653288358"
Oct 30 19:43:04.210403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.210339    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-xxgh4" podStartSLOduration=2.475036311 podStartE2EDuration="6.21031886s" totalImagesPullingTime="3.735282549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:04.207592777 +0000 UTC m=+508.708296500" watchObservedRunningTime="2026-10-30 19:43:04.21031886 +0000 UTC m=+508.711022585"
Oct 30 19:43:04.252595 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:04.252529    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn" podStartSLOduration=2.4950889849999998 podStartE2EDuration="6.252512048s" totalImagesPullingTime="3.757423063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:04.250510305 +0000 UTC m=+508.751214026" watchObservedRunningTime="2026-10-30 19:43:04.252512048 +0000 UTC m=+508.753215772"
Oct 30 19:43:15.163886 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:15.163832    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-xxgh4"
Oct 30 19:43:20.902373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:20.902325    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsc9q"]
Oct 30 19:43:20.906129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:20.906099    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsc9q"
Oct 30 19:43:20.924217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:20.924179    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsc9q"]
Oct 30 19:43:20.933184 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:20.933151    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhvjj\""
Oct 30 19:43:21.028859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.028822    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc92k\" (UniqueName: \"kubernetes.io/projected/2433a5bb-124f-42b9-9a77-27874d0f8dec-kube-api-access-lc92k\") pod \"authorino-6f6944bcdd-rsc9q\" (UID: \"2433a5bb-124f-42b9-9a77-27874d0f8dec\") " pod="kuadrant-system/authorino-6f6944bcdd-rsc9q"
Oct 30 19:43:21.071926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.071885    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsc9q"]
Oct 30 19:43:21.072145 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:43:21.072114    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lc92k], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-rsc9q" podUID="2433a5bb-124f-42b9-9a77-27874d0f8dec"
Oct 30 19:43:21.104773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.104725    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2sd5h"]
Oct 30 19:43:21.108405 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.108375    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:21.129446 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.129411    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc92k\" (UniqueName: \"kubernetes.io/projected/2433a5bb-124f-42b9-9a77-27874d0f8dec-kube-api-access-lc92k\") pod \"authorino-6f6944bcdd-rsc9q\" (UID: \"2433a5bb-124f-42b9-9a77-27874d0f8dec\") " pod="kuadrant-system/authorino-6f6944bcdd-rsc9q"
Oct 30 19:43:21.129874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.129854    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2sd5h"]
Oct 30 19:43:21.154002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.153839    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc92k\" (UniqueName: \"kubernetes.io/projected/2433a5bb-124f-42b9-9a77-27874d0f8dec-kube-api-access-lc92k\") pod \"authorino-6f6944bcdd-rsc9q\" (UID: \"2433a5bb-124f-42b9-9a77-27874d0f8dec\") " pod="kuadrant-system/authorino-6f6944bcdd-rsc9q"
Oct 30 19:43:21.211152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.211109    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:43:21.223303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.223264    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsc9q"
Oct 30 19:43:21.230389 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.230345    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-tls-cert\") pod \"authorino-6889cbd8c4-2sd5h\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") " pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:21.230591 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.230484    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngrwq\" (UniqueName: \"kubernetes.io/projected/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-kube-api-access-ngrwq\") pod \"authorino-6889cbd8c4-2sd5h\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") " pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:21.246521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.246458    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsc9q"
Oct 30 19:43:21.278698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.278660    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2sd5h"]
Oct 30 19:43:21.278918 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:43:21.278900    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ngrwq tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-2sd5h" podUID="278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f"
Oct 30 19:43:21.313526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.313460    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68c4854845-lg9t7"]
Oct 30 19:43:21.322079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.322038    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.331433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.331388    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2433a5bb-124f-42b9-9a77-27874d0f8dec-kube-api-access-lc92k\" (UniqueName: \"kubernetes.io/projected/2433a5bb-124f-42b9-9a77-27874d0f8dec-kube-api-access-lc92k\") pod \"2433a5bb-124f-42b9-9a77-27874d0f8dec\" (UID: \"2433a5bb-124f-42b9-9a77-27874d0f8dec\") "
Oct 30 19:43:21.331697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.331668    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-tls-cert\") pod \"authorino-6889cbd8c4-2sd5h\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") " pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:21.331833 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.331781    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngrwq\" (UniqueName: \"kubernetes.io/projected/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-kube-api-access-ngrwq\") pod \"authorino-6889cbd8c4-2sd5h\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") " pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:21.334770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.334692    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2433a5bb-124f-42b9-9a77-27874d0f8dec-kube-api-access-lc92k" pod "2433a5bb-124f-42b9-9a77-27874d0f8dec" (UID: "2433a5bb-124f-42b9-9a77-27874d0f8dec"). InnerVolumeSpecName "kube-api-access-lc92k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:21.335095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.335064    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-tls-cert\") pod \"authorino-6889cbd8c4-2sd5h\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") " pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:21.338268 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.338233    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68c4854845-lg9t7"]
Oct 30 19:43:21.352543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.352498    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngrwq\" (UniqueName: \"kubernetes.io/projected/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-kube-api-access-ngrwq\") pod \"authorino-6889cbd8c4-2sd5h\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") " pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:21.432385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.432345    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d2d7c8e-7890-4c4d-ba38-199264815fb5-tls-cert\") pod \"authorino-68c4854845-lg9t7\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") " pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.432601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.432433    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjjl\" (UniqueName: \"kubernetes.io/projected/1d2d7c8e-7890-4c4d-ba38-199264815fb5-kube-api-access-mdjjl\") pod \"authorino-68c4854845-lg9t7\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") " pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.432601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.432574    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc92k\" (UniqueName: \"kubernetes.io/projected/2433a5bb-124f-42b9-9a77-27874d0f8dec-kube-api-access-lc92k\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:43:21.533794 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.533754    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d2d7c8e-7890-4c4d-ba38-199264815fb5-tls-cert\") pod \"authorino-68c4854845-lg9t7\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") " pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.533995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.533842    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjjl\" (UniqueName: \"kubernetes.io/projected/1d2d7c8e-7890-4c4d-ba38-199264815fb5-kube-api-access-mdjjl\") pod \"authorino-68c4854845-lg9t7\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") " pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.536335 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.536303    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d2d7c8e-7890-4c4d-ba38-199264815fb5-tls-cert\") pod \"authorino-68c4854845-lg9t7\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") " pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.555931 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.555898    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjjl\" (UniqueName: \"kubernetes.io/projected/1d2d7c8e-7890-4c4d-ba38-199264815fb5-kube-api-access-mdjjl\") pod \"authorino-68c4854845-lg9t7\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") " pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.660835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.641280    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:43:21.952021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:21.951989    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68c4854845-lg9t7"]
Oct 30 19:43:21.952486 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:43:21.952450    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d2d7c8e_7890_4c4d_ba38_199264815fb5.slice/crio-b459994938304fa9c3120df892b46cd2da563dbf7fd8972c080c5c010b0a00dc WatchSource:0}: Error finding container b459994938304fa9c3120df892b46cd2da563dbf7fd8972c080c5c010b0a00dc: Status 404 returned error can't find the container with id b459994938304fa9c3120df892b46cd2da563dbf7fd8972c080c5c010b0a00dc
Oct 30 19:43:22.230460 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.230357    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c4854845-lg9t7" event={"ID":"1d2d7c8e-7890-4c4d-ba38-199264815fb5","Type":"ContainerStarted","Data":"b459994938304fa9c3120df892b46cd2da563dbf7fd8972c080c5c010b0a00dc"}
Oct 30 19:43:22.232238 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.232208    4519 generic.go:356] "Generic (PLEG): container finished" podID="924bd04d-c63b-441d-a17b-fff0e36df522" containerID="e864fa5d7153466d4142734fe14f3f69056fe2b9e66c47364081885c3c751d65" exitCode=0
Oct 30 19:43:22.232375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.232269    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:22.232375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.232278    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" event={"ID":"924bd04d-c63b-441d-a17b-fff0e36df522","Type":"ContainerDied","Data":"e864fa5d7153466d4142734fe14f3f69056fe2b9e66c47364081885c3c751d65"}
Oct 30 19:43:22.232375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.232321    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsc9q"
Oct 30 19:43:22.232808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.232790    4519 scope.go:122] "RemoveContainer" containerID="e864fa5d7153466d4142734fe14f3f69056fe2b9e66c47364081885c3c751d65"
Oct 30 19:43:22.233915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.233883    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:22.234004 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.233978    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:22.234044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.234028    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:22.290698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.290655    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:22.440970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.440934    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsc9q"]
Oct 30 19:43:22.441228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.441095    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-tls-cert\" (UniqueName: \"kubernetes.io/secret/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-tls-cert\") pod \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") "
Oct 30 19:43:22.441373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.441347    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-kube-api-access-ngrwq\" (UniqueName: \"kubernetes.io/projected/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-kube-api-access-ngrwq\") pod \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\" (UID: \"278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f\") "
Oct 30 19:43:22.443885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.443837    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-tls-cert" pod "278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f" (UID: "278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:22.444378 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.444347    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-kube-api-access-ngrwq" pod "278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f" (UID: "278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f"). InnerVolumeSpecName "kube-api-access-ngrwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:22.464047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.464007    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsc9q"]
Oct 30 19:43:22.542708 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.542451    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngrwq\" (UniqueName: \"kubernetes.io/projected/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-kube-api-access-ngrwq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:43:22.542708 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:22.542563    4519 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f-tls-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:43:23.241327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.241235    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" event={"ID":"924bd04d-c63b-441d-a17b-fff0e36df522","Type":"ContainerStarted","Data":"373f6fc4f5281b8d293c8da87608c8e68a057424a2375d46abe9090d889323c7"}
Oct 30 19:43:23.243230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.243194    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2sd5h"
Oct 30 19:43:23.243365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.243196    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c4854845-lg9t7" event={"ID":"1d2d7c8e-7890-4c4d-ba38-199264815fb5","Type":"ContainerStarted","Data":"7a343ee2fd9172482fe21bfed1da360c2d90d4a43babe7f2b006fe9c79d0dcfb"}
Oct 30 19:43:23.335648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.335584    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68c4854845-lg9t7" podStartSLOduration=1.315565271 podStartE2EDuration="2.335566169s" totalImagesPullingTime="1.020000898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:23.333881415 +0000 UTC m=+527.834585244" watchObservedRunningTime="2026-10-30 19:43:23.335566169 +0000 UTC m=+527.836269891"
Oct 30 19:43:23.417504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.416957    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2sd5h"]
Oct 30 19:43:23.426618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.426581    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2sd5h"]
Oct 30 19:43:23.612286 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.612201    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2433a5bb-124f-42b9-9a77-27874d0f8dec" path="/var/lib/kubelet/pods/2433a5bb-124f-42b9-9a77-27874d0f8dec/volumes"
Oct 30 19:43:23.612584 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.612568    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f" path="/var/lib/kubelet/pods/278ed2ee-65cd-4acb-aea0-f8cdf9a4cc2f/volumes"
Oct 30 19:43:23.896381 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.896269    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:43:23.896587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.896532    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-7vbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:43:23.896661 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:23.896591    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" podUID="924bd04d-c63b-441d-a17b-fff0e36df522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:43:24.895976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:24.895934    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-7vbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:43:24.896384 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:24.895996    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" podUID="924bd04d-c63b-441d-a17b-fff0e36df522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:43:25.896806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:25.896760    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-7vbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:43:25.897223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:25.896821    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" podUID="924bd04d-c63b-441d-a17b-fff0e36df522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:43:26.896454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:26.896417    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-7vbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:43:26.896671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:26.896508    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" podUID="924bd04d-c63b-441d-a17b-fff0e36df522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:43:27.896193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:27.896159    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-7vbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 19:43:27.896724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:27.896690    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" podUID="924bd04d-c63b-441d-a17b-fff0e36df522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 19:43:28.899915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:28.899878    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:43:28.900423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:28.900275    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:43:28.901119 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:28.901100    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7"
Oct 30 19:43:32.220682 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:43:32.220642    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:43:35.123146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:35.123110    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:43:35.126134 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:43:35.126109    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vmkgb"
Oct 30 19:44:02.220660 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:44:02.220614    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:44:04.741167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.741128    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"]
Oct 30 19:44:04.744975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.744949    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:04.766036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.765997    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"]
Oct 30 19:44:04.827801 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.827760    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9a4b1c7-eb14-43dd-9b71-325a66bef4f8-cert\") pod \"llmisvc-controller-manager-6466c554d8-t6gkp\" (UID: \"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:04.828012 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.827827    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrvx\" (UniqueName: \"kubernetes.io/projected/d9a4b1c7-eb14-43dd-9b71-325a66bef4f8-kube-api-access-rtrvx\") pod \"llmisvc-controller-manager-6466c554d8-t6gkp\" (UID: \"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:04.929406 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.929354    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9a4b1c7-eb14-43dd-9b71-325a66bef4f8-cert\") pod \"llmisvc-controller-manager-6466c554d8-t6gkp\" (UID: \"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:04.929641 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.929446    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrvx\" (UniqueName: \"kubernetes.io/projected/d9a4b1c7-eb14-43dd-9b71-325a66bef4f8-kube-api-access-rtrvx\") pod \"llmisvc-controller-manager-6466c554d8-t6gkp\" (UID: \"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:04.932086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.932056    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9a4b1c7-eb14-43dd-9b71-325a66bef4f8-cert\") pod \"llmisvc-controller-manager-6466c554d8-t6gkp\" (UID: \"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:04.951197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:04.951160    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrvx\" (UniqueName: \"kubernetes.io/projected/d9a4b1c7-eb14-43dd-9b71-325a66bef4f8-kube-api-access-rtrvx\") pod \"llmisvc-controller-manager-6466c554d8-t6gkp\" (UID: \"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:05.058001 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:05.057906    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:05.368099 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:05.368012    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"]
Oct 30 19:44:05.368388 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:44:05.368363    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a4b1c7_eb14_43dd_9b71_325a66bef4f8.slice/crio-69c3ab4600b9a361937c5ae55bb6fb4476ba09525dad5f784ac80da41c9e55d0 WatchSource:0}: Error finding container 69c3ab4600b9a361937c5ae55bb6fb4476ba09525dad5f784ac80da41c9e55d0: Status 404 returned error can't find the container with id 69c3ab4600b9a361937c5ae55bb6fb4476ba09525dad5f784ac80da41c9e55d0
Oct 30 19:44:05.485626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:05.485582    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp" event={"ID":"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8","Type":"ContainerStarted","Data":"69c3ab4600b9a361937c5ae55bb6fb4476ba09525dad5f784ac80da41c9e55d0"}
Oct 30 19:44:06.492728 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:06.492690    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp" event={"ID":"d9a4b1c7-eb14-43dd-9b71-325a66bef4f8","Type":"ContainerStarted","Data":"bd2f4dea00eed60f87fe5f8de69d56e14173a64a77062ac1f1f94a75dcbba071"}
Oct 30 19:44:06.493138 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:06.492821    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:06.537713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:06.537654    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp" podStartSLOduration=2.204915684 podStartE2EDuration="2.537640232s" totalImagesPullingTime="332.724548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:06.535987729 +0000 UTC m=+571.036691454" watchObservedRunningTime="2026-10-30 19:44:06.537640232 +0000 UTC m=+571.038344022"
Oct 30 19:44:32.221677 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:44:32.221581    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:44:37.499580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.499548    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-t6gkp"
Oct 30 19:44:37.573034 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.572998    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"]
Oct 30 19:44:37.573270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.573241    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn" podUID="39adede1-163e-45df-b381-ab419e3a433a" containerName="manager" containerID="cri-o://db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe" gracePeriod=30
Oct 30 19:44:37.829284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.829253    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:44:37.933989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.933947    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39adede1-163e-45df-b381-ab419e3a433a-cert\" (UniqueName: \"kubernetes.io/secret/39adede1-163e-45df-b381-ab419e3a433a-cert\") pod \"39adede1-163e-45df-b381-ab419e3a433a\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") "
Oct 30 19:44:37.934172 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.934051    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39adede1-163e-45df-b381-ab419e3a433a-kube-api-access-4ss6v\" (UniqueName: \"kubernetes.io/projected/39adede1-163e-45df-b381-ab419e3a433a-kube-api-access-4ss6v\") pod \"39adede1-163e-45df-b381-ab419e3a433a\" (UID: \"39adede1-163e-45df-b381-ab419e3a433a\") "
Oct 30 19:44:37.936210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.936176    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39adede1-163e-45df-b381-ab419e3a433a-cert" pod "39adede1-163e-45df-b381-ab419e3a433a" (UID: "39adede1-163e-45df-b381-ab419e3a433a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:37.936210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:37.936189    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39adede1-163e-45df-b381-ab419e3a433a-kube-api-access-4ss6v" pod "39adede1-163e-45df-b381-ab419e3a433a" (UID: "39adede1-163e-45df-b381-ab419e3a433a"). InnerVolumeSpecName "kube-api-access-4ss6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:38.035082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.035043    4519 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39adede1-163e-45df-b381-ab419e3a433a-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:44:38.035082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.035078    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ss6v\" (UniqueName: \"kubernetes.io/projected/39adede1-163e-45df-b381-ab419e3a433a-kube-api-access-4ss6v\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:44:38.662816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.662781    4519 generic.go:356] "Generic (PLEG): container finished" podID="39adede1-163e-45df-b381-ab419e3a433a" containerID="db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe" exitCode=0
Oct 30 19:44:38.663273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.662833    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"
Oct 30 19:44:38.663273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.662865    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn" event={"ID":"39adede1-163e-45df-b381-ab419e3a433a","Type":"ContainerDied","Data":"db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe"}
Oct 30 19:44:38.663273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.662924    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn" event={"ID":"39adede1-163e-45df-b381-ab419e3a433a","Type":"ContainerDied","Data":"bc447726fd3c69f41ef215d8383ea5e91fad7d0ce843736c82f4eeb6377ddf8a"}
Oct 30 19:44:38.663273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.662954    4519 scope.go:122] "RemoveContainer" containerID="db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe"
Oct 30 19:44:38.672583 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.672562    4519 scope.go:122] "RemoveContainer" containerID="db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe"
Oct 30 19:44:38.672936 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:44:38.672912    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe\": container with ID starting with db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe not found: ID does not exist" containerID="db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe"
Oct 30 19:44:38.673002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.672949    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe"} err="failed to get container status \"db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe\": rpc error: code = NotFound desc = could not find container \"db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe\": container with ID starting with db313a337a8058bd619137224604c27297c249cff04e9c0dfe2cd22b0a633afe not found: ID does not exist"
Oct 30 19:44:38.722722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.722685    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"]
Oct 30 19:44:38.735044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:38.735006    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-m4hnn"]
Oct 30 19:44:39.611177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:44:39.611139    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39adede1-163e-45df-b381-ab419e3a433a" path="/var/lib/kubelet/pods/39adede1-163e-45df-b381-ab419e3a433a/volumes"
Oct 30 19:45:02.226118 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:45:02.226073    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:45:07.836142 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:07.836040    4519 generic.go:356] "Generic (PLEG): container finished" podID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerID="f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69" exitCode=1
Oct 30 19:45:07.836142 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:07.836118    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" event={"ID":"ae9324bb-be21-4f9f-a5cd-a983a2d585aa","Type":"ContainerDied","Data":"f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69"}
Oct 30 19:45:07.836660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:07.836528    4519 scope.go:122] "RemoveContainer" containerID="f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69"
Oct 30 19:45:08.848221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:08.848178    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" event={"ID":"ae9324bb-be21-4f9f-a5cd-a983a2d585aa","Type":"ContainerStarted","Data":"2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180"}
Oct 30 19:45:08.849088 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:08.849066    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:45:19.859321 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:19.859280    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 19:45:29.819983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:29.819940    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68c4854845-lg9t7"]
Oct 30 19:45:29.820575 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:29.820148    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68c4854845-lg9t7" podUID="1d2d7c8e-7890-4c4d-ba38-199264815fb5" containerName="authorino" containerID="cri-o://7a343ee2fd9172482fe21bfed1da360c2d90d4a43babe7f2b006fe9c79d0dcfb" gracePeriod=30
Oct 30 19:45:29.976556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:29.976515    4519 generic.go:356] "Generic (PLEG): container finished" podID="1d2d7c8e-7890-4c4d-ba38-199264815fb5" containerID="7a343ee2fd9172482fe21bfed1da360c2d90d4a43babe7f2b006fe9c79d0dcfb" exitCode=0
Oct 30 19:45:29.976864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:29.976556    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c4854845-lg9t7" event={"ID":"1d2d7c8e-7890-4c4d-ba38-199264815fb5","Type":"ContainerDied","Data":"7a343ee2fd9172482fe21bfed1da360c2d90d4a43babe7f2b006fe9c79d0dcfb"}
Oct 30 19:45:30.076084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.075998    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:45:30.142590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.142540    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2d7c8e-7890-4c4d-ba38-199264815fb5-kube-api-access-mdjjl\" (UniqueName: \"kubernetes.io/projected/1d2d7c8e-7890-4c4d-ba38-199264815fb5-kube-api-access-mdjjl\") pod \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") "
Oct 30 19:45:30.142808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.142617    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2d7c8e-7890-4c4d-ba38-199264815fb5-tls-cert\" (UniqueName: \"kubernetes.io/secret/1d2d7c8e-7890-4c4d-ba38-199264815fb5-tls-cert\") pod \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\" (UID: \"1d2d7c8e-7890-4c4d-ba38-199264815fb5\") "
Oct 30 19:45:30.144915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.144878    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2d7c8e-7890-4c4d-ba38-199264815fb5-kube-api-access-mdjjl" pod "1d2d7c8e-7890-4c4d-ba38-199264815fb5" (UID: "1d2d7c8e-7890-4c4d-ba38-199264815fb5"). InnerVolumeSpecName "kube-api-access-mdjjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:30.154200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.154163    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2d7c8e-7890-4c4d-ba38-199264815fb5-tls-cert" pod "1d2d7c8e-7890-4c4d-ba38-199264815fb5" (UID: "1d2d7c8e-7890-4c4d-ba38-199264815fb5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:30.243924 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.243868    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdjjl\" (UniqueName: \"kubernetes.io/projected/1d2d7c8e-7890-4c4d-ba38-199264815fb5-kube-api-access-mdjjl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:45:30.243924 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.243906    4519 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d2d7c8e-7890-4c4d-ba38-199264815fb5-tls-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:45:30.987194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.987150    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c4854845-lg9t7"
Oct 30 19:45:30.987729 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.987145    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c4854845-lg9t7" event={"ID":"1d2d7c8e-7890-4c4d-ba38-199264815fb5","Type":"ContainerDied","Data":"b459994938304fa9c3120df892b46cd2da563dbf7fd8972c080c5c010b0a00dc"}
Oct 30 19:45:30.987729 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:30.987306    4519 scope.go:122] "RemoveContainer" containerID="7a343ee2fd9172482fe21bfed1da360c2d90d4a43babe7f2b006fe9c79d0dcfb"
Oct 30 19:45:31.082107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:31.082050    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68c4854845-lg9t7"]
Oct 30 19:45:31.092355 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:31.092310    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68c4854845-lg9t7"]
Oct 30 19:45:31.615031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:31.614985    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2d7c8e-7890-4c4d-ba38-199264815fb5" path="/var/lib/kubelet/pods/1d2d7c8e-7890-4c4d-ba38-199264815fb5/volumes"
Oct 30 19:45:32.221573 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:45:32.221537    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:45:37.592534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.592488    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"]
Oct 30 19:45:37.593073 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.593052    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1d2d7c8e-7890-4c4d-ba38-199264815fb5" containerName="authorino"
Oct 30 19:45:37.593115 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.593077    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2d7c8e-7890-4c4d-ba38-199264815fb5" containerName="authorino"
Oct 30 19:45:37.593155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.593115    4519 cpu_manager.go:414] "containerMap: removing container" podUID="39adede1-163e-45df-b381-ab419e3a433a" containerName="manager"
Oct 30 19:45:37.593155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.593123    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39adede1-163e-45df-b381-ab419e3a433a" containerName="manager"
Oct 30 19:45:37.593225 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.593213    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="39adede1-163e-45df-b381-ab419e3a433a" containerName="manager"
Oct 30 19:45:37.593256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.593232    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2d7c8e-7890-4c4d-ba38-199264815fb5" containerName="authorino"
Oct 30 19:45:37.598810 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.598783    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.620344 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.620305    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"]
Oct 30 19:45:37.711910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.711865    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.711910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.711904    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.712196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.711999    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.712196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.712082    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.712196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.712112    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.712370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.712233    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.712370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.712272    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.712370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.712297    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.712370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.712348    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pldsk\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-kube-api-access-pldsk\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813194    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813229    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813249    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813289    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pldsk\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-kube-api-access-pldsk\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813336    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813363    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813397    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813453    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813510    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813911 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813722    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.813911 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813781    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.814010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.813998    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.814067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.814023    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.814235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.814217    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.815754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.815732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.815986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.815965    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.833158 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.833123    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.833317 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.833247    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pldsk\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-kube-api-access-pldsk\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nxd25\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:37.914293 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:37.914254    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:38.235401 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:45:38.235361    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e6714c_8a9c_4eb6_9495_2bb6f35e270f.slice/crio-c8185492fe4ff0f47de4bd089c511692017e79a470a3b5f2e0e50c2483ee0c58 WatchSource:0}: Error finding container c8185492fe4ff0f47de4bd089c511692017e79a470a3b5f2e0e50c2483ee0c58: Status 404 returned error can't find the container with id c8185492fe4ff0f47de4bd089c511692017e79a470a3b5f2e0e50c2483ee0c58
Oct 30 19:45:38.235730 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:38.235704    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"]
Oct 30 19:45:38.237688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:38.237653    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:38.237821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:38.237775    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:38.237885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:38.237820    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:39.050493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:39.048701    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" event={"ID":"46e6714c-8a9c-4eb6-9495-2bb6f35e270f","Type":"ContainerStarted","Data":"67ca2b70de02ca50190765b5761b4bf1841734db240fd4efb1c155491208cd2b"}
Oct 30 19:45:39.050493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:39.048775    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" event={"ID":"46e6714c-8a9c-4eb6-9495-2bb6f35e270f","Type":"ContainerStarted","Data":"c8185492fe4ff0f47de4bd089c511692017e79a470a3b5f2e0e50c2483ee0c58"}
Oct 30 19:45:39.123458 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:39.123400    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podStartSLOduration=2.123384511 podStartE2EDuration="2.123384511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:39.122896197 +0000 UTC m=+663.623599923" watchObservedRunningTime="2026-10-30 19:45:39.123384511 +0000 UTC m=+663.624088237"
Oct 30 19:45:39.915377 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:39.915338    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:39.916811 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:39.916784    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nxd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:45:39.916937 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:39.916835    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 19:45:40.915584 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:40.915543    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nxd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:45:40.916013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:40.915605    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 19:45:41.915745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:41.915705    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nxd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:45:41.916172 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:41.915768    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 19:45:42.915631 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:42.915595    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nxd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:45:42.915837 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:42.915652    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 19:45:43.915169 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:43.915132    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nxd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:45:43.915370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:43.915191    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 19:45:44.915215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:44.915168    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nxd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:45:44.915669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:44.915244    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 19:45:46.633912 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:46.633874    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:46.634452 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:46.634428    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:45:46.635776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:45:46.635755    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:46:02.221404 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:46:02.221366    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:46:26.536872 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.536828    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"]
Oct 30 19:46:26.542335 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.542300    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.551296 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.550950    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"]
Oct 30 19:46:26.557909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.557881    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6c4dd1-openshift-default-dockercfg-tw2l4\""
Oct 30 19:46:26.692354 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.692311    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.692696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.692678    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.693298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.693274    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.693539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.693522    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.693705 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.693683    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.693850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.693723    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.693850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.693770    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.693850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.693796    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cv6s\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-kube-api-access-6cv6s\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.693850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.693843    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795079    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795124    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795147    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795168    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795203    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795226    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cv6s\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-kube-api-access-6cv6s\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795303    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795652    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795703    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795735    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795762    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795766    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.795910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.795872    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.796459 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.796437    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.799025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.798686    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.799025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.798755    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.829910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.829877    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.830098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.830073    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cv6s\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-kube-api-access-6cv6s\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:26.878993 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:26.878950    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:27.237948 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:46:27.237904    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5f5b1e_6a46_4dc9_bd0e_839f710e3674.slice/crio-18bdacedb97939ac5756d1d2a8385678e3dba59a0987eda46e655aa178469344 WatchSource:0}: Error finding container 18bdacedb97939ac5756d1d2a8385678e3dba59a0987eda46e655aa178469344: Status 404 returned error can't find the container with id 18bdacedb97939ac5756d1d2a8385678e3dba59a0987eda46e655aa178469344
Oct 30 19:46:27.238311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:27.238286    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"]
Oct 30 19:46:27.240666 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:27.240631    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:27.240747 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:27.240706    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:27.240747 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:27.240736    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:27.332327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:27.332280    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" event={"ID":"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674","Type":"ContainerStarted","Data":"a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b"}
Oct 30 19:46:27.332533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:27.332363    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" event={"ID":"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674","Type":"ContainerStarted","Data":"18bdacedb97939ac5756d1d2a8385678e3dba59a0987eda46e655aa178469344"}
Oct 30 19:46:27.879980 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:27.879946    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:28.896590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:28.896546    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:28.967905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:28.967831    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podStartSLOduration=2.967812228 podStartE2EDuration="2.967812228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:27.403899369 +0000 UTC m=+711.904603095" watchObservedRunningTime="2026-10-30 19:46:28.967812228 +0000 UTC m=+713.468516035"
Oct 30 19:46:29.351603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:29.351566    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:29.352752 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:29.352726    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:32.223667 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:46:32.223609    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:46:36.374949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.374910    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"]
Oct 30 19:46:36.378853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.378828    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:36.396435 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.396397    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-109cfe\""
Oct 30 19:46:36.405612 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.405575    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-109cfe-dockercfg-698c2\""
Oct 30 19:46:36.407917 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.407884    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"]
Oct 30 19:46:36.501301 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.501262    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flpld\" (UniqueName: \"kubernetes.io/projected/dae99e72-5490-4883-be29-e73f0f85d920-kube-api-access-flpld\") pod \"payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:36.501589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.501447    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae99e72-5490-4883-be29-e73f0f85d920-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:36.602878 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.602841    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpld\" (UniqueName: \"kubernetes.io/projected/dae99e72-5490-4883-be29-e73f0f85d920-kube-api-access-flpld\") pod \"payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:36.603109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.602939    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae99e72-5490-4883-be29-e73f0f85d920-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:36.603667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.603636    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae99e72-5490-4883-be29-e73f0f85d920-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:36.623636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.623599    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpld\" (UniqueName: \"kubernetes.io/projected/dae99e72-5490-4883-be29-e73f0f85d920-kube-api-access-flpld\") pod \"payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:36.692878 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:36.692839    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:37.007589 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:46:37.007545    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae99e72_5490_4883_be29_e73f0f85d920.slice/crio-094095b4a57994a99ba9979e0f66c2ac1d59ea35848ce7aecce358e8b981e60d WatchSource:0}: Error finding container 094095b4a57994a99ba9979e0f66c2ac1d59ea35848ce7aecce358e8b981e60d: Status 404 returned error can't find the container with id 094095b4a57994a99ba9979e0f66c2ac1d59ea35848ce7aecce358e8b981e60d
Oct 30 19:46:37.007748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:37.007683    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"]
Oct 30 19:46:37.009972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:37.009951    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:46:37.403857 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:37.403771    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" event={"ID":"dae99e72-5490-4883-be29-e73f0f85d920","Type":"ContainerStarted","Data":"094095b4a57994a99ba9979e0f66c2ac1d59ea35848ce7aecce358e8b981e60d"}
Oct 30 19:46:38.209102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.209066    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"]
Oct 30 19:46:38.213364 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.213326    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.239219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.239179    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"]
Oct 30 19:46:38.247103 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.247067    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8a3426-dockercfg-7q528\""
Oct 30 19:46:38.276519 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.276448    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8a3426\""
Oct 30 19:46:38.322949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.322912    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtf94\" (UniqueName: \"kubernetes.io/projected/2a524b77-1bd9-45cd-98d9-699ade653c7c-kube-api-access-mtf94\") pod \"payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.323143 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.323059    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a524b77-1bd9-45cd-98d9-699ade653c7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.424633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.424593    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a524b77-1bd9-45cd-98d9-699ade653c7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.425134 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.424710    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtf94\" (UniqueName: \"kubernetes.io/projected/2a524b77-1bd9-45cd-98d9-699ade653c7c-kube-api-access-mtf94\") pod \"payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.425380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.425349    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a524b77-1bd9-45cd-98d9-699ade653c7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.447136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.447099    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtf94\" (UniqueName: \"kubernetes.io/projected/2a524b77-1bd9-45cd-98d9-699ade653c7c-kube-api-access-mtf94\") pod \"payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.530727 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.530407    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:38.870166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:38.858975    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"]
Oct 30 19:46:39.418442 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:39.418401    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" event={"ID":"2a524b77-1bd9-45cd-98d9-699ade653c7c","Type":"ContainerStarted","Data":"612eed5b785129ce710e1fa5dab2a36e00d49de8eef98e30e12714651d012508"}
Oct 30 19:46:39.420229 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:39.420193    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" event={"ID":"dae99e72-5490-4883-be29-e73f0f85d920","Type":"ContainerStarted","Data":"02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69"}
Oct 30 19:46:39.420398 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:39.420373    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:39.463026 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:39.462950    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" podStartSLOduration=1.205695793 podStartE2EDuration="3.462925748s" totalImagesPullingTime="2.257229955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:39.459690908 +0000 UTC m=+723.960394635" watchObservedRunningTime="2026-10-30 19:46:39.462925748 +0000 UTC m=+723.963629475"
Oct 30 19:46:40.427207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:40.427163    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" event={"ID":"2a524b77-1bd9-45cd-98d9-699ade653c7c","Type":"ContainerStarted","Data":"05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774"}
Oct 30 19:46:40.470750 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:40.470668    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" podStartSLOduration=1.92901528 podStartE2EDuration="2.47064832s" totalImagesPullingTime="541.63304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:40.468994869 +0000 UTC m=+724.969698593" watchObservedRunningTime="2026-10-30 19:46:40.47064832 +0000 UTC m=+724.971352045"
Oct 30 19:46:41.433110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:41.433076    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:47.039826 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.039691    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"]
Oct 30 19:46:47.043892 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.043863    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.084814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.084776    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"default-dockercfg-9dr8w\""
Oct 30 19:46:47.085055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.084802    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"kube-root-ca.crt\""
Oct 30 19:46:47.085154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.084865    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"simulator-w6-f349a1-kserve-self-signed-certs\""
Oct 30 19:46:47.085230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.084915    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"openshift-service-ca.crt\""
Oct 30 19:46:47.127563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.127516    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"]
Oct 30 19:46:47.224590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.224550    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ff036e-01cc-42be-8059-e4721ce56fbc-tls-certs\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.224772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.224604    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74krn\" (UniqueName: \"kubernetes.io/projected/77ff036e-01cc-42be-8059-e4721ce56fbc-kube-api-access-74krn\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.224772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.224722    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-model-cache\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.224772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.224756    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-tmp-dir\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.224909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.224791    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-home\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.224909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.224855    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-dshm\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.326211 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326110    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ff036e-01cc-42be-8059-e4721ce56fbc-tls-certs\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.326211 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326159    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74krn\" (UniqueName: \"kubernetes.io/projected/77ff036e-01cc-42be-8059-e4721ce56fbc-kube-api-access-74krn\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.326211 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-model-cache\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.326517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326227    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-tmp-dir\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.326517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-home\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.326517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326432    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-dshm\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.326710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326688    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-tmp-dir\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.327019 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326968    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-home\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.327019 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.326979    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-model-cache\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.329686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.329659    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-dshm\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.329914 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.329891    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ff036e-01cc-42be-8059-e4721ce56fbc-tls-certs\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.357918 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.357843    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74krn\" (UniqueName: \"kubernetes.io/projected/77ff036e-01cc-42be-8059-e4721ce56fbc-kube-api-access-74krn\") pod \"simulator-w6-f349a1-kserve-75864459cf-4fddm\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") " pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.388973 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.388668    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:47.759987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:47.759921    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"]
Oct 30 19:46:48.512549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:48.512503    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" event={"ID":"77ff036e-01cc-42be-8059-e4721ce56fbc","Type":"ContainerStarted","Data":"19cf5db916698a079f3ba2e77474c43209c39b4e294736a01ca9f0f7a122d921"}
Oct 30 19:46:50.430998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:50.430969    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 19:46:50.529312 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:50.529272    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" event={"ID":"77ff036e-01cc-42be-8059-e4721ce56fbc","Type":"ContainerStarted","Data":"cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c"}
Oct 30 19:46:50.571260 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:50.571177    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podStartSLOduration=2.79495637 podStartE2EDuration="4.571156638s" totalImagesPullingTime="1.776200268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:50.569398561 +0000 UTC m=+735.070102285" watchObservedRunningTime="2026-10-30 19:46:50.571156638 +0000 UTC m=+735.071860363"
Oct 30 19:46:51.390050 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:51.390012    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:52.443544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:52.443510    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 19:46:52.544659 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:52.544625    4519 generic.go:356] "Generic (PLEG): container finished" podID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerID="a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b" exitCode=0
Oct 30 19:46:52.544834 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:52.544707    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" event={"ID":"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674","Type":"ContainerDied","Data":"a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b"}
Oct 30 19:46:52.545254 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:52.545232    4519 scope.go:122] "RemoveContainer" containerID="a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b"
Oct 30 19:46:52.546735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:52.546705    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:52.546831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:52.546794    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:52.546896 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:52.546839    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:53.563490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:53.562877    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" event={"ID":"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674","Type":"ContainerStarted","Data":"d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742"}
Oct 30 19:46:53.879793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:53.879661    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:53.880040 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:53.879886    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:46:53.880040 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:53.879943    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:46:54.880024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:54.879985    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:46:54.880448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:54.880043    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:46:55.884407 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:55.883886    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:46:55.884407 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:55.883951    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:46:56.560490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.557568    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"]
Oct 30 19:46:56.568487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.567692    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.586713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.585957    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-01f7d3-openshift-default-dockercfg-mt2rl\""
Oct 30 19:46:56.607660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.607603    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"]
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672007    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672071    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672134    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672216    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672251    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7tn\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-kube-api-access-jd7tn\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672284    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672313    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.672491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.672425    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb07fbb-698c-459b-a21b-130779cc9afd-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775077    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7tn\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-kube-api-access-jd7tn\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775171    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775218    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775278    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb07fbb-698c-459b-a21b-130779cc9afd-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775420    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775508    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775563    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.776490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.775601    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.779907 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.779874    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb07fbb-698c-459b-a21b-130779cc9afd-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.781964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.781900    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.782282 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.782242    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.782571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.782549    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.782822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.782802    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.785535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.784448    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.787007 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.786940    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.796180 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.796148    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.798547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.798517    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7tn\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-kube-api-access-jd7tn\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:56.881553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.880911    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:46:56.881553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.880982    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:46:56.912308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:56.912270    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:57.286455 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.285787    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"]
Oct 30 19:46:57.290117 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:46:57.289287    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb07fbb_698c_459b_a21b_130779cc9afd.slice/crio-80a07bd11541ff11c459a89b7e913e70bee667d12891e7e4251083f5d9375e48 WatchSource:0}: Error finding container 80a07bd11541ff11c459a89b7e913e70bee667d12891e7e4251083f5d9375e48: Status 404 returned error can't find the container with id 80a07bd11541ff11c459a89b7e913e70bee667d12891e7e4251083f5d9375e48
Oct 30 19:46:57.292502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.292033    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:57.292502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.292133    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:57.292502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.292213    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:57.389928 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.389896    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:57.402740 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.402707    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:57.603553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.603426    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" event={"ID":"7eb07fbb-698c-459b-a21b-130779cc9afd","Type":"ContainerStarted","Data":"78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd"}
Oct 30 19:46:57.603553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.603518    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" event={"ID":"7eb07fbb-698c-459b-a21b-130779cc9afd","Type":"ContainerStarted","Data":"80a07bd11541ff11c459a89b7e913e70bee667d12891e7e4251083f5d9375e48"}
Oct 30 19:46:57.616855 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.616820    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:46:57.729025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.728914    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podStartSLOduration=1.728895059 podStartE2EDuration="1.728895059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:57.676705861 +0000 UTC m=+742.177409599" watchObservedRunningTime="2026-10-30 19:46:57.728895059 +0000 UTC m=+742.229598789"
Oct 30 19:46:57.882217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.882126    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:46:57.882836 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.882424    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:46:57.912642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:57.912590    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:46:58.914987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:58.914951    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:46:58.915437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:58.915012    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:46:59.216716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.216632    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:59.217314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.217287    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:59.218302 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.218285    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 19:46:59.584180 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.584065    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"]
Oct 30 19:46:59.594180 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.594142    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.609421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.609388    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"simulator-w2-284bf6-kserve-self-signed-certs\""
Oct 30 19:46:59.609626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.609523    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"kube-root-ca.crt\""
Oct 30 19:46:59.609626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.609579    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"default-dockercfg-ppjjf\""
Oct 30 19:46:59.609626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.609616    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"openshift-service-ca.crt\""
Oct 30 19:46:59.623079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.623046    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"]
Oct 30 19:46:59.729830 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.729785    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4b2k\" (UniqueName: \"kubernetes.io/projected/a25a5898-dfb6-4257-935d-e4290aa91888-kube-api-access-g4b2k\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.730375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.730347    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-tmp-dir\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.730556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.730398    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-home\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.730626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.730536    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25a5898-dfb6-4257-935d-e4290aa91888-tls-certs\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.730626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.730590    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-dshm\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.730733 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.730701    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-model-cache\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.831852 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.831811    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4b2k\" (UniqueName: \"kubernetes.io/projected/a25a5898-dfb6-4257-935d-e4290aa91888-kube-api-access-g4b2k\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.831912    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-tmp-dir\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.831942    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-home\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.831979    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25a5898-dfb6-4257-935d-e4290aa91888-tls-certs\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.832016    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-dshm\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832290 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.832093    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-model-cache\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832447 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.832415    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-home\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.832484    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-tmp-dir\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.832610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.832567    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-model-cache\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.834552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.834491    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-dshm\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.834807 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.834787    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25a5898-dfb6-4257-935d-e4290aa91888-tls-certs\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.864510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.864419    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4b2k\" (UniqueName: \"kubernetes.io/projected/a25a5898-dfb6-4257-935d-e4290aa91888-kube-api-access-g4b2k\") pod \"simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") " pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:46:59.913255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.913218    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:46:59.913437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.913280    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:46:59.921648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:46:59.921611    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:47:00.274951 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:00.274914    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"]
Oct 30 19:47:00.276051 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:47:00.276024    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25a5898_dfb6_4257_935d_e4290aa91888.slice/crio-9c2f0f2f0375e0935fc54212711632535a83ce79e7f1ff0242fe1fe1f82eb5b0 WatchSource:0}: Error finding container 9c2f0f2f0375e0935fc54212711632535a83ce79e7f1ff0242fe1fe1f82eb5b0: Status 404 returned error can't find the container with id 9c2f0f2f0375e0935fc54212711632535a83ce79e7f1ff0242fe1fe1f82eb5b0
Oct 30 19:47:00.672501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:00.671544    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" event={"ID":"a25a5898-dfb6-4257-935d-e4290aa91888","Type":"ContainerStarted","Data":"9590bfa4a5c5602002f0a77ff8a461d61ea9c5fac43c0e011ac945f2b838b77b"}
Oct 30 19:47:00.672501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:00.671605    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" event={"ID":"a25a5898-dfb6-4257-935d-e4290aa91888","Type":"ContainerStarted","Data":"9c2f0f2f0375e0935fc54212711632535a83ce79e7f1ff0242fe1fe1f82eb5b0"}
Oct 30 19:47:00.716561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:00.716481    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podStartSLOduration=1.716445399 podStartE2EDuration="1.716445399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:00.714701761 +0000 UTC m=+745.215405486" watchObservedRunningTime="2026-10-30 19:47:00.716445399 +0000 UTC m=+745.217149136"
Oct 30 19:47:00.913256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:00.913162    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:47:00.913545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:00.913419    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:47:00.923279 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:00.923172    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:47:01.913242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:01.913130    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:47:01.913599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:01.913570    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:47:02.227561 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:47:02.227434    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:47:02.913314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:02.913284    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:47:02.913663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:02.913639    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:47:04.028153 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:04.028122    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:47:04.028935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:04.028913    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:47:04.029911 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:04.029859    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 19:47:06.625689 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.625650    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"]
Oct 30 19:47:06.636068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.636027    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.652596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.652556    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"]
Oct 30 19:47:06.661621 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.661586    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"default-dockercfg-5xd58\""
Oct 30 19:47:06.661834 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.661714    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"simulator-w5-6c4dd1-kserve-self-signed-certs\""
Oct 30 19:47:06.665018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.664918    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"openshift-service-ca.crt\""
Oct 30 19:47:06.665018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.664961    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"kube-root-ca.crt\""
Oct 30 19:47:06.740063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.740027    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a974082e-4a02-4a95-9a91-a2f97228a9f2-tls-certs\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.740270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.740084    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-model-cache\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.740270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.740177    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-home\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.740371 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.740310    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-tmp-dir\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.740427 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.740376    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh578\" (UniqueName: \"kubernetes.io/projected/a974082e-4a02-4a95-9a91-a2f97228a9f2-kube-api-access-xh578\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.740516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.740477    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-dshm\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841538 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841451    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-model-cache\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841567    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-home\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841605    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-tmp-dir\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841646    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh578\" (UniqueName: \"kubernetes.io/projected/a974082e-4a02-4a95-9a91-a2f97228a9f2-kube-api-access-xh578\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841665    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-dshm\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841724    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a974082e-4a02-4a95-9a91-a2f97228a9f2-tls-certs\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841945    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-model-cache\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.841998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.841961    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-home\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.842171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.842151    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-tmp-dir\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.844124 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.844091    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-dshm\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.845441 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.845419    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a974082e-4a02-4a95-9a91-a2f97228a9f2-tls-certs\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.918078 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.918047    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh578\" (UniqueName: \"kubernetes.io/projected/a974082e-4a02-4a95-9a91-a2f97228a9f2-kube-api-access-xh578\") pod \"simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") " pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:06.951103 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:06.951064    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:07.285611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:07.285575    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"]
Oct 30 19:47:07.285611 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:47:07.285592    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda974082e_4a02_4a95_9a91_a2f97228a9f2.slice/crio-62a92535b3069663b27deb831f985b0afd0bbe33c423b9e6efdc0d78abf63c78 WatchSource:0}: Error finding container 62a92535b3069663b27deb831f985b0afd0bbe33c423b9e6efdc0d78abf63c78: Status 404 returned error can't find the container with id 62a92535b3069663b27deb831f985b0afd0bbe33c423b9e6efdc0d78abf63c78
Oct 30 19:47:07.734000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:07.733959    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" event={"ID":"a974082e-4a02-4a95-9a91-a2f97228a9f2","Type":"ContainerStarted","Data":"86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200"}
Oct 30 19:47:07.734000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:07.734005    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" event={"ID":"a974082e-4a02-4a95-9a91-a2f97228a9f2","Type":"ContainerStarted","Data":"62a92535b3069663b27deb831f985b0afd0bbe33c423b9e6efdc0d78abf63c78"}
Oct 30 19:47:07.785483 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:07.785400    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podStartSLOduration=1.7853766690000001 podStartE2EDuration="1.785376669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:07.784539815 +0000 UTC m=+752.285243537" watchObservedRunningTime="2026-10-30 19:47:07.785376669 +0000 UTC m=+752.286080394"
Oct 30 19:47:07.951919 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:07.951878    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:09.922775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:09.922732    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:47:09.937331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:09.937296    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:47:10.769410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:10.769369    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:47:14.112599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.112528    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"]
Oct 30 19:47:14.117587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.117540    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.128264 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.128227    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tmp-dir\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.128448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.128280    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tls-certs\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.128561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.128443    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-home\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.128561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.128509    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmmf\" (UniqueName: \"kubernetes.io/projected/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-kube-api-access-hcmmf\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.128670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.128565    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-dshm\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.128670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.128590    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-model-cache\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.151690 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.151649    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"kube-root-ca.crt\""
Oct 30 19:47:14.151690 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.151685    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"openshift-service-ca.crt\""
Oct 30 19:47:14.151942 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.151769    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"simulator-w3-01f7d3-kserve-self-signed-certs\""
Oct 30 19:47:14.152343 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.152297    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"default-dockercfg-8cntf\""
Oct 30 19:47:14.154388 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.154360    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"]
Oct 30 19:47:14.229686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.229649    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tmp-dir\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.229686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.229694    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tls-certs\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.229975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.229757    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-home\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.229975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.229783    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmmf\" (UniqueName: \"kubernetes.io/projected/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-kube-api-access-hcmmf\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.229975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.229814    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-dshm\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.229975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.229837    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-model-cache\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.230196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.230165    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tmp-dir\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.230258 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.230235    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-model-cache\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.230941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.230916    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-home\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.233008 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.232983    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tls-certs\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.233728 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.233697    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-dshm\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.254639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.254606    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmmf\" (UniqueName: \"kubernetes.io/projected/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-kube-api-access-hcmmf\") pod \"simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") " pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.431967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.431932    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:14.745428 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.745341    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"]
Oct 30 19:47:14.751286 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:47:14.751251    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef7e3f0b_4ff8_4a13_a928_8b99655a00e9.slice/crio-69cfd2f10edc334a89a2fcafa193533432ea42d48d42b5f1d37cc733b7626bc8 WatchSource:0}: Error finding container 69cfd2f10edc334a89a2fcafa193533432ea42d48d42b5f1d37cc733b7626bc8: Status 404 returned error can't find the container with id 69cfd2f10edc334a89a2fcafa193533432ea42d48d42b5f1d37cc733b7626bc8
Oct 30 19:47:14.777988 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:14.777947    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" event={"ID":"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9","Type":"ContainerStarted","Data":"69cfd2f10edc334a89a2fcafa193533432ea42d48d42b5f1d37cc733b7626bc8"}
Oct 30 19:47:15.785728 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:15.785688    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" event={"ID":"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9","Type":"ContainerStarted","Data":"7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812"}
Oct 30 19:47:15.830359 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:15.830290    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podStartSLOduration=1.830271816 podStartE2EDuration="1.830271816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:15.827942971 +0000 UTC m=+760.328646696" watchObservedRunningTime="2026-10-30 19:47:15.830271816 +0000 UTC m=+760.330975539"
Oct 30 19:47:16.432835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:16.432789    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:16.951303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:16.951259    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:16.965934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:16.965900    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:17.810603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:17.810570    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:47:24.432402 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:24.432308    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:24.446515 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:24.446461    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:24.859434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:24.859345    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:47:26.934714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.934671    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"]
Oct 30 19:47:26.938625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.938595    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:26.966308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.965627    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"default-dockercfg-ljjjh\""
Oct 30 19:47:26.966308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.965733    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"simulator-w4-8a3426-kserve-self-signed-certs\""
Oct 30 19:47:26.966308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.965763    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"openshift-service-ca.crt\""
Oct 30 19:47:26.967699 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.967620    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"kube-root-ca.crt\""
Oct 30 19:47:26.970669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.970639    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"]
Oct 30 19:47:26.971237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.971214    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-home\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:26.971338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.971272    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02d3d7d-3a67-4934-995f-12773e9a5874-tls-certs\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:26.971439 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.971415    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-dshm\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:26.971519 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.971505    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2jj\" (UniqueName: \"kubernetes.io/projected/f02d3d7d-3a67-4934-995f-12773e9a5874-kube-api-access-cw2jj\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:26.971589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.971575    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-model-cache\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:26.971655 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:26.971601    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-tmp-dir\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.072924 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.072870    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-home\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.072960    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02d3d7d-3a67-4934-995f-12773e9a5874-tls-certs\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.073018    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-dshm\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.073083    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2jj\" (UniqueName: \"kubernetes.io/projected/f02d3d7d-3a67-4934-995f-12773e9a5874-kube-api-access-cw2jj\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.073146    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-model-cache\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.073174    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-tmp-dir\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073415 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.073364    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-home\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.073728    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-model-cache\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.073903 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.073845    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-tmp-dir\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.076305 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.076276    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02d3d7d-3a67-4934-995f-12773e9a5874-tls-certs\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.076442 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.076300    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-dshm\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.094571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.094538    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2jj\" (UniqueName: \"kubernetes.io/projected/f02d3d7d-3a67-4934-995f-12773e9a5874-kube-api-access-cw2jj\") pod \"simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") " pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.256633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.256533    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:27.569874 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:47:27.569838    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf02d3d7d_3a67_4934_995f_12773e9a5874.slice/crio-520acbf5888bc55e3f0f5479135a9089ac2279566a37cccf818ba49258daadde WatchSource:0}: Error finding container 520acbf5888bc55e3f0f5479135a9089ac2279566a37cccf818ba49258daadde: Status 404 returned error can't find the container with id 520acbf5888bc55e3f0f5479135a9089ac2279566a37cccf818ba49258daadde
Oct 30 19:47:27.570976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.570944    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"]
Oct 30 19:47:27.868768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.868663    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" event={"ID":"f02d3d7d-3a67-4934-995f-12773e9a5874","Type":"ContainerStarted","Data":"8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b"}
Oct 30 19:47:27.868768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.868710    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" event={"ID":"f02d3d7d-3a67-4934-995f-12773e9a5874","Type":"ContainerStarted","Data":"520acbf5888bc55e3f0f5479135a9089ac2279566a37cccf818ba49258daadde"}
Oct 30 19:47:27.915774 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:27.915708    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podStartSLOduration=1.915689688 podStartE2EDuration="1.915689688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:27.913833352 +0000 UTC m=+772.414537075" watchObservedRunningTime="2026-10-30 19:47:27.915689688 +0000 UTC m=+772.416393413"
Oct 30 19:47:28.256964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:28.256919    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:32.221299 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:47:32.221259    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:47:37.256981 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:37.256935    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:37.275768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:37.275732    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:37.943029 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:37.942999    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:47:39.066592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.066557    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"]
Oct 30 19:47:39.070634 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.070608    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.092441 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.092400    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"]
Oct 30 19:47:39.099592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.099546    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"premium-w2-284bf6-kserve-self-signed-certs\""
Oct 30 19:47:39.202185 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.202145    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714c801b-8384-450a-bef2-53760f5efd2a-tls-certs\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.202416 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.202235    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-model-cache\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.202416 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.202343    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklfh\" (UniqueName: \"kubernetes.io/projected/714c801b-8384-450a-bef2-53760f5efd2a-kube-api-access-qklfh\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.202567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.202443    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-tmp-dir\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.202567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.202536    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-home\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.202661 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.202575    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-dshm\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.303613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.303568    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-home\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.303613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.303620    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-dshm\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.303897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.303694    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714c801b-8384-450a-bef2-53760f5efd2a-tls-certs\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.303897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.303767    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-model-cache\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.303897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.303831    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklfh\" (UniqueName: \"kubernetes.io/projected/714c801b-8384-450a-bef2-53760f5efd2a-kube-api-access-qklfh\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.303897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.303876    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-tmp-dir\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.304105 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.304075    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-home\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.304191 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.304161    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-tmp-dir\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.304430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.304391    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-model-cache\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.307067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.307024    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-dshm\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.307220 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.307093    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714c801b-8384-450a-bef2-53760f5efd2a-tls-certs\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.325905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.325819    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklfh\" (UniqueName: \"kubernetes.io/projected/714c801b-8384-450a-bef2-53760f5efd2a-kube-api-access-qklfh\") pod \"premium-w2-284bf6-kserve-5795d7cb-5zpqc\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") " pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.384957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.384888    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:39.711779 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.711732    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"]
Oct 30 19:47:39.949579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.949539    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" event={"ID":"714c801b-8384-450a-bef2-53760f5efd2a","Type":"ContainerStarted","Data":"4f92532538612876c8b604a2fcfb8a8c134ea860ea7b543e4f985ff5e679ca6f"}
Oct 30 19:47:39.949734 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.949587    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" event={"ID":"714c801b-8384-450a-bef2-53760f5efd2a","Type":"ContainerStarted","Data":"e24d33da6a2b5a3bcc8949382de1b0164534a9bf5e60a6f303e11aab843d6385"}
Oct 30 19:47:39.995857 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:39.995737    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podStartSLOduration=0.995721508 podStartE2EDuration="995.721508ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:39.994754895 +0000 UTC m=+784.495458622" watchObservedRunningTime="2026-10-30 19:47:39.995721508 +0000 UTC m=+784.496425257"
Oct 30 19:47:40.386050 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:40.385957    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:41.011934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.011891    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"]
Oct 30 19:47:41.016793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.016758    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.031698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.031666    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6c4dd1\"/\"premium-w5-6c4dd1-kserve-self-signed-certs\""
Oct 30 19:47:41.039110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.039078    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"]
Oct 30 19:47:41.122874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.122834    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-home\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.122874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.122879    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tmp-dir\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.123107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.122970    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdcnh\" (UniqueName: \"kubernetes.io/projected/8a1f16b8-14c6-4dc5-83fd-901a42afade4-kube-api-access-zdcnh\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.123181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.123159    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tls-certs\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.123223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.123200    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-dshm\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.123316 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.123299    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-model-cache\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.224494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.224324    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-dshm\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.224494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.224411    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-model-cache\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.224831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.224552    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-home\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.224831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.224580    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tmp-dir\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.224831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.224616    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdcnh\" (UniqueName: \"kubernetes.io/projected/8a1f16b8-14c6-4dc5-83fd-901a42afade4-kube-api-access-zdcnh\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.224831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.224758    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tls-certs\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.234808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.225691    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-home\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.234808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.225743    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tmp-dir\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.234808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.225788    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-model-cache\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.234808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.228524    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tls-certs\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.234808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.228733    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-dshm\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.252002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.251949    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdcnh\" (UniqueName: \"kubernetes.io/projected/8a1f16b8-14c6-4dc5-83fd-901a42afade4-kube-api-access-zdcnh\") pod \"premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") " pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.329892 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.329786    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:41.379224 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.379186    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"]
Oct 30 19:47:41.389507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.383862    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.403241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.403200    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-284bf6-openshift-default-dockercfg-fq2xw\""
Oct 30 19:47:41.411480 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.411430    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"]
Oct 30 19:47:41.527906 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.527872    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.527906 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.527908    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntm9c\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-kube-api-access-ntm9c\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.528166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.527938    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.528166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.528083    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6ba7b-880c-4d52-946a-30d883491b8b-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.528166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.528111    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.528166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.528140    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.528330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.528272    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.528330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.528327    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.528419 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.528356    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629183 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629089    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629183 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629127    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntm9c\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-kube-api-access-ntm9c\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629242    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629275    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6ba7b-880c-4d52-946a-30d883491b8b-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629297    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629327    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629397    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629458    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629536    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629611    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629712    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629859    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.629895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.629877    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.630033 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.630014    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6ba7b-880c-4d52-946a-30d883491b8b-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.631850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.631826    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.632122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.632103    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.644911 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.644881    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"]
Oct 30 19:47:41.645591 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:47:41.645557    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a1f16b8_14c6_4dc5_83fd_901a42afade4.slice/crio-a135e3549bb4101317c5d79e303757174e3d22d762df2d6abb098a3a7231c4d6 WatchSource:0}: Error finding container a135e3549bb4101317c5d79e303757174e3d22d762df2d6abb098a3a7231c4d6: Status 404 returned error can't find the container with id a135e3549bb4101317c5d79e303757174e3d22d762df2d6abb098a3a7231c4d6
Oct 30 19:47:41.649705 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.649678    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntm9c\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-kube-api-access-ntm9c\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.649705 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.649690    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.711420 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.711383    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:41.964113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.964068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" event={"ID":"8a1f16b8-14c6-4dc5-83fd-901a42afade4","Type":"ContainerStarted","Data":"b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b"}
Oct 30 19:47:41.964113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:41.964122    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" event={"ID":"8a1f16b8-14c6-4dc5-83fd-901a42afade4","Type":"ContainerStarted","Data":"a135e3549bb4101317c5d79e303757174e3d22d762df2d6abb098a3a7231c4d6"}
Oct 30 19:47:42.011023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.010947    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podStartSLOduration=2.010924363 podStartE2EDuration="2.010924363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:42.006644935 +0000 UTC m=+786.507348640" watchObservedRunningTime="2026-10-30 19:47:42.010924363 +0000 UTC m=+786.511628092"
Oct 30 19:47:42.038496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.038450    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"]
Oct 30 19:47:42.038662 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:47:42.038639    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d6ba7b_880c_4d52_946a_30d883491b8b.slice/crio-dc71a85ec93a4a27b55e70882ed2cfaf8d6e3bdb393d560f884e9fc70b754bae WatchSource:0}: Error finding container dc71a85ec93a4a27b55e70882ed2cfaf8d6e3bdb393d560f884e9fc70b754bae: Status 404 returned error can't find the container with id dc71a85ec93a4a27b55e70882ed2cfaf8d6e3bdb393d560f884e9fc70b754bae
Oct 30 19:47:42.041070 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.041034    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:47:42.041171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.041113    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:47:42.041171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.041156    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:47:42.330166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.330055    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:42.996436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.996390    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" event={"ID":"d9d6ba7b-880c-4d52-946a-30d883491b8b","Type":"ContainerStarted","Data":"777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f"}
Oct 30 19:47:42.999774 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:42.997604    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" event={"ID":"d9d6ba7b-880c-4d52-946a-30d883491b8b","Type":"ContainerStarted","Data":"dc71a85ec93a4a27b55e70882ed2cfaf8d6e3bdb393d560f884e9fc70b754bae"}
Oct 30 19:47:43.062966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:43.062895    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podStartSLOduration=2.062871706 podStartE2EDuration="2.062871706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:43.060107741 +0000 UTC m=+787.560811462" watchObservedRunningTime="2026-10-30 19:47:43.062871706 +0000 UTC m=+787.563575436"
Oct 30 19:47:43.711748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:43.711702    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:43.713564 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:43.713537    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:47:43.713695 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:43.713587    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:47:44.712255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:44.712217    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:47:44.712696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:44.712278    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:47:45.712512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:45.712456    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:47:45.712935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:45.712534    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:47:46.712330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:46.712287    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:47:46.712538 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:46.712359    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:47:47.711880 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:47.711841    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:47:47.712093 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:47.711909    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:47:49.385085 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:49.385041    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:49.403403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:49.403366    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:49.480677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:49.477639    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:49.480677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:49.478501    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:49.480677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:49.479762    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 19:47:50.058981 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:50.058948    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:47:51.330909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:51.330872    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:51.344195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:51.344162    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:52.067281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:52.067244    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:47:56.541166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.541125    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"]
Oct 30 19:47:56.544406 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.544386    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.566133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.566093    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f349a1-openshift-default-dockercfg-n7zxx\""
Oct 30 19:47:56.578232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.578176    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"]
Oct 30 19:47:56.623276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623240    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623286    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstz9\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-kube-api-access-hstz9\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623413    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623478    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623521    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623624    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623655    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623672    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.623934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.623771    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.724972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.724928    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.724972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.724976    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725076    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstz9\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-kube-api-access-hstz9\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725134    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725182    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725340 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725206    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725350    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725380    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725424    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725447    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725706 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725601    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725836    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.725926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.726010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.725985    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.727545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.727517    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.727787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.727768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.762820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.762784    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.765005 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.764959    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstz9\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-kube-api-access-hstz9\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:56.860776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:56.860657    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:57.209514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:57.209485    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"]
Oct 30 19:47:57.212286 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:47:57.212258    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb3f0f5_e299_4472_aa62_6936c4cf8bbc.slice/crio-c335e3dd67e189f52652ff397d6ff4fe56530a63567cda87bd6070d5d1676f85 WatchSource:0}: Error finding container c335e3dd67e189f52652ff397d6ff4fe56530a63567cda87bd6070d5d1676f85: Status 404 returned error can't find the container with id c335e3dd67e189f52652ff397d6ff4fe56530a63567cda87bd6070d5d1676f85
Oct 30 19:47:57.215087 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:57.215046    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:47:57.216563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:57.215132    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:47:57.216563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:57.215173    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:47:58.099785 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:58.099746    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" event={"ID":"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc","Type":"ContainerStarted","Data":"4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2"}
Oct 30 19:47:58.100204 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:58.099804    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" event={"ID":"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc","Type":"ContainerStarted","Data":"c335e3dd67e189f52652ff397d6ff4fe56530a63567cda87bd6070d5d1676f85"}
Oct 30 19:47:58.186501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:58.186398    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podStartSLOduration=2.186377441 podStartE2EDuration="2.186377441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:58.16701647 +0000 UTC m=+802.667720197" watchObservedRunningTime="2026-10-30 19:47:58.186377441 +0000 UTC m=+802.687081164"
Oct 30 19:47:58.869495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:58.861865    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:47:58.869495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:58.867290    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:47:58.869495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:58.867353    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:47:59.869490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:59.861894    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:47:59.869490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:47:59.861963    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:48:00.862931 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:00.862888    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:48:00.863283 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:00.863250    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:48:01.865492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:01.861699    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:48:01.865492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:01.861771    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:48:02.222833 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:48:02.222784    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:48:02.867493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:02.862122    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:48:02.867493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:02.862189    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:48:03.861953 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:03.861911    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:48:03.862184 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:03.861983    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:48:04.866695 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:04.866663    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:48:04.867224 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:04.867082    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:48:04.867932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:04.867910    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 19:48:12.247044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.247007    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"]
Oct 30 19:48:12.252151 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.252124    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.270292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.270265    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-109cfe-openshift-default-dockercfg-4jtdp\""
Oct 30 19:48:12.290298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290259    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290300    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290321    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290348    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176a9878-2bb1-444d-a865-9149b3061fd1-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290540    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290752 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290725    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp925\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-kube-api-access-xp925\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290795    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176a9878-2bb1-444d-a865-9149b3061fd1-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290824    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.290921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.290863    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.295349 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.295314    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"]
Oct 30 19:48:12.392397 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392360    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176a9878-2bb1-444d-a865-9149b3061fd1-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392403    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392441    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392511    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392529    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392547    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392565    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176a9878-2bb1-444d-a865-9149b3061fd1-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392924 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392654    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.392924 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392740    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp925\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-kube-api-access-xp925\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.393036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.392974    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.393348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.393316    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.393490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.393348    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176a9878-2bb1-444d-a865-9149b3061fd1-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.393490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.393420    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.393490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.393443    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.395118 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.395093    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.395270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.395151    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176a9878-2bb1-444d-a865-9149b3061fd1-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.413793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.413763    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.413965 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.413930    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp925\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-kube-api-access-xp925\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.572202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.572102    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:12.909622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.909587    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"]
Oct 30 19:48:12.909819 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:12.909798    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176a9878_2bb1_444d_a865_9149b3061fd1.slice/crio-f740761865d3cf2d7af6beba3f788ffe7f03351454c2caddb4a62339486422b6 WatchSource:0}: Error finding container f740761865d3cf2d7af6beba3f788ffe7f03351454c2caddb4a62339486422b6: Status 404 returned error can't find the container with id f740761865d3cf2d7af6beba3f788ffe7f03351454c2caddb4a62339486422b6
Oct 30 19:48:12.912561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.912528    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:12.912671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.912593    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:12.912671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:12.912621    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:13.202068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:13.202029    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" event={"ID":"176a9878-2bb1-444d-a865-9149b3061fd1","Type":"ContainerStarted","Data":"a1b587dea3e2a0c95487ba83306ab7cfd0cfd242db67f34125f74f44159f3610"}
Oct 30 19:48:13.202257 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:13.202088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" event={"ID":"176a9878-2bb1-444d-a865-9149b3061fd1","Type":"ContainerStarted","Data":"f740761865d3cf2d7af6beba3f788ffe7f03351454c2caddb4a62339486422b6"}
Oct 30 19:48:13.249514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:13.249349    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podStartSLOduration=1.249329827 podStartE2EDuration="1.249329827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:13.248963091 +0000 UTC m=+817.749666816" watchObservedRunningTime="2026-10-30 19:48:13.249329827 +0000 UTC m=+817.750033552"
Oct 30 19:48:13.574865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:13.574696    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:13.577612 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:13.577573    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:13.577793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:13.577639    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:14.578054 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:14.578011    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:14.578549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:14.578083    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:15.577491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:15.575882    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:15.577491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:15.575953    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:16.573490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:16.572893    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:16.573490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:16.572960    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:17.573968 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:17.573920    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:17.574627 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:17.574593    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:18.577916 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:18.577873    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:18.578592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:18.578554    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:19.572696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:19.572651    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:19.572898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:19.572727    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:20.577635 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:20.577603    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:20.578154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:20.578035    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:20.578798 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:20.578780    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 19:48:21.579362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.579316    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"]
Oct 30 19:48:21.585164 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.585136    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.603535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.603501    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"]
Oct 30 19:48:21.613486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.613248    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:48:21.613486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.613348    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:21.722714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.722659    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.722952 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.722923    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwzc9\" (UniqueName: \"kubernetes.io/projected/6451a9a8-6593-4f80-be6f-3d6b082fafab-kube-api-access-wwzc9\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.723047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.723018    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.824595 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.824550    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.824755 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.824617    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.824755 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.824729    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzc9\" (UniqueName: \"kubernetes.io/projected/6451a9a8-6593-4f80-be6f-3d6b082fafab-kube-api-access-wwzc9\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.825380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.825353    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.827195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.827166    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.844710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.844634    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzc9\" (UniqueName: \"kubernetes.io/projected/6451a9a8-6593-4f80-be6f-3d6b082fafab-kube-api-access-wwzc9\") pod \"maas-api-revoke-sub-fe8950b1-dgkxp\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:21.900268 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:21.900222    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:22.208698 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:22.208662    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6451a9a8_6593_4f80_be6f_3d6b082fafab.slice/crio-8def332adc0bb41e341bd46dcd86466f46eaf63e6d9ba1e9e1226ee85751ae6d WatchSource:0}: Error finding container 8def332adc0bb41e341bd46dcd86466f46eaf63e6d9ba1e9e1226ee85751ae6d: Status 404 returned error can't find the container with id 8def332adc0bb41e341bd46dcd86466f46eaf63e6d9ba1e9e1226ee85751ae6d
Oct 30 19:48:22.210676 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:22.209950    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"]
Oct 30 19:48:22.290765 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:22.290719    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp" event={"ID":"6451a9a8-6593-4f80-be6f-3d6b082fafab","Type":"ContainerStarted","Data":"8def332adc0bb41e341bd46dcd86466f46eaf63e6d9ba1e9e1226ee85751ae6d"}
Oct 30 19:48:25.315152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:25.315058    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp" event={"ID":"6451a9a8-6593-4f80-be6f-3d6b082fafab","Type":"ContainerStarted","Data":"a3a10093ac4c8fc5f7d0ea12dc5662496d6cd8dca396e79f8ba72a038283eaf0"}
Oct 30 19:48:25.356616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:25.356548    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp" podStartSLOduration=1.8995724649999999 podStartE2EDuration="4.356526271s" totalImagesPullingTime="2.456953806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:25.355186776 +0000 UTC m=+829.855890501" watchObservedRunningTime="2026-10-30 19:48:25.356526271 +0000 UTC m=+829.857229997"
Oct 30 19:48:26.322573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:26.322537    4519 generic.go:356] "Generic (PLEG): container finished" podID="6451a9a8-6593-4f80-be6f-3d6b082fafab" containerID="a3a10093ac4c8fc5f7d0ea12dc5662496d6cd8dca396e79f8ba72a038283eaf0" exitCode=0
Oct 30 19:48:26.322995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:26.322629    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp" event={"ID":"6451a9a8-6593-4f80-be6f-3d6b082fafab","Type":"ContainerDied","Data":"a3a10093ac4c8fc5f7d0ea12dc5662496d6cd8dca396e79f8ba72a038283eaf0"}
Oct 30 19:48:27.492654 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.492620    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:27.594667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.594512    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6451a9a8-6593-4f80-be6f-3d6b082fafab-kube-api-access-wwzc9\" (UniqueName: \"kubernetes.io/projected/6451a9a8-6593-4f80-be6f-3d6b082fafab-kube-api-access-wwzc9\") pod \"6451a9a8-6593-4f80-be6f-3d6b082fafab\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") "
Oct 30 19:48:27.595108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.595075    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-xks-ca\") pod \"6451a9a8-6593-4f80-be6f-3d6b082fafab\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") "
Oct 30 19:48:27.595242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.595125    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-openshift-ca\") pod \"6451a9a8-6593-4f80-be6f-3d6b082fafab\" (UID: \"6451a9a8-6593-4f80-be6f-3d6b082fafab\") "
Oct 30 19:48:27.595848 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.595822    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-openshift-ca" pod "6451a9a8-6593-4f80-be6f-3d6b082fafab" (UID: "6451a9a8-6593-4f80-be6f-3d6b082fafab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:27.598069 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.598036    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6451a9a8-6593-4f80-be6f-3d6b082fafab-kube-api-access-wwzc9" pod "6451a9a8-6593-4f80-be6f-3d6b082fafab" (UID: "6451a9a8-6593-4f80-be6f-3d6b082fafab"). InnerVolumeSpecName "kube-api-access-wwzc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:27.598069 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.598047    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-xks-ca" pod "6451a9a8-6593-4f80-be6f-3d6b082fafab" (UID: "6451a9a8-6593-4f80-be6f-3d6b082fafab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:27.697136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.697095    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwzc9\" (UniqueName: \"kubernetes.io/projected/6451a9a8-6593-4f80-be6f-3d6b082fafab-kube-api-access-wwzc9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:27.697136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.697129    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:27.697136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:27.697145    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6451a9a8-6593-4f80-be6f-3d6b082fafab-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:28.336783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.336752    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"
Oct 30 19:48:28.336978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.336753    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp" event={"ID":"6451a9a8-6593-4f80-be6f-3d6b082fafab","Type":"ContainerDied","Data":"8def332adc0bb41e341bd46dcd86466f46eaf63e6d9ba1e9e1226ee85751ae6d"}
Oct 30 19:48:28.336978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.336877    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8def332adc0bb41e341bd46dcd86466f46eaf63e6d9ba1e9e1226ee85751ae6d"
Oct 30 19:48:28.585218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.585182    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"]
Oct 30 19:48:28.585675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.585660    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6451a9a8-6593-4f80-be6f-3d6b082fafab" containerName="revoke-keys"
Oct 30 19:48:28.585718 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.585676    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6451a9a8-6593-4f80-be6f-3d6b082fafab" containerName="revoke-keys"
Oct 30 19:48:28.585783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.585773    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6451a9a8-6593-4f80-be6f-3d6b082fafab" containerName="revoke-keys"
Oct 30 19:48:28.589180 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.589136    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.607359 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.607319    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.607644 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.607621    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfztx\" (UniqueName: \"kubernetes.io/projected/5eca6708-9134-4cde-ad50-7129cac56ea4-kube-api-access-hfztx\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.607971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.607950    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.611605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.610951    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:28.611605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.611313    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:48:28.622666 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.622603    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"]
Oct 30 19:48:28.709084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.709041    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.709084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.709092    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.709350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.709116    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfztx\" (UniqueName: \"kubernetes.io/projected/5eca6708-9134-4cde-ad50-7129cac56ea4-kube-api-access-hfztx\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.709942 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.709910    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.711730 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.711704    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.733330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.733298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfztx\" (UniqueName: \"kubernetes.io/projected/5eca6708-9134-4cde-ad50-7129cac56ea4-kube-api-access-hfztx\") pod \"maas-api-revoke-sub-3130ac33-c7q42\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:28.902499 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:28.902372    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:29.210811 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:29.210769    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eca6708_9134_4cde_ad50_7129cac56ea4.slice/crio-208a2fb3512357e19db16d82073a7b7f74348418dc027d05e5b3f5fd7f3bbf4c WatchSource:0}: Error finding container 208a2fb3512357e19db16d82073a7b7f74348418dc027d05e5b3f5fd7f3bbf4c: Status 404 returned error can't find the container with id 208a2fb3512357e19db16d82073a7b7f74348418dc027d05e5b3f5fd7f3bbf4c
Oct 30 19:48:29.210811 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:29.210787    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"]
Oct 30 19:48:29.345295 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:29.345253    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42" event={"ID":"5eca6708-9134-4cde-ad50-7129cac56ea4","Type":"ContainerStarted","Data":"208a2fb3512357e19db16d82073a7b7f74348418dc027d05e5b3f5fd7f3bbf4c"}
Oct 30 19:48:30.353589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:30.353544    4519 generic.go:356] "Generic (PLEG): container finished" podID="5eca6708-9134-4cde-ad50-7129cac56ea4" containerID="e8c387b5628fd834c5973c489b297cdea99d726505c947d1bec322432607accb" exitCode=0
Oct 30 19:48:30.354214 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:30.353626    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42" event={"ID":"5eca6708-9134-4cde-ad50-7129cac56ea4","Type":"ContainerDied","Data":"e8c387b5628fd834c5973c489b297cdea99d726505c947d1bec322432607accb"}
Oct 30 19:48:31.512109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.512077    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:31.542475 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.542440    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-xks-ca\") pod \"5eca6708-9134-4cde-ad50-7129cac56ea4\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") "
Oct 30 19:48:31.542625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.542584    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eca6708-9134-4cde-ad50-7129cac56ea4-kube-api-access-hfztx\" (UniqueName: \"kubernetes.io/projected/5eca6708-9134-4cde-ad50-7129cac56ea4-kube-api-access-hfztx\") pod \"5eca6708-9134-4cde-ad50-7129cac56ea4\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") "
Oct 30 19:48:31.542625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.542614    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-openshift-ca\") pod \"5eca6708-9134-4cde-ad50-7129cac56ea4\" (UID: \"5eca6708-9134-4cde-ad50-7129cac56ea4\") "
Oct 30 19:48:31.542989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.542961    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-openshift-ca" pod "5eca6708-9134-4cde-ad50-7129cac56ea4" (UID: "5eca6708-9134-4cde-ad50-7129cac56ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:31.544646 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.544622    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-xks-ca" pod "5eca6708-9134-4cde-ad50-7129cac56ea4" (UID: "5eca6708-9134-4cde-ad50-7129cac56ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:31.544742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.544723    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eca6708-9134-4cde-ad50-7129cac56ea4-kube-api-access-hfztx" pod "5eca6708-9134-4cde-ad50-7129cac56ea4" (UID: "5eca6708-9134-4cde-ad50-7129cac56ea4"). InnerVolumeSpecName "kube-api-access-hfztx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:31.644021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.643985    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:31.644021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.644016    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfztx\" (UniqueName: \"kubernetes.io/projected/5eca6708-9134-4cde-ad50-7129cac56ea4-kube-api-access-hfztx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:31.644021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:31.644026    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eca6708-9134-4cde-ad50-7129cac56ea4-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:32.222290 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:48:32.222253    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:48:32.370396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:32.370359    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42" event={"ID":"5eca6708-9134-4cde-ad50-7129cac56ea4","Type":"ContainerDied","Data":"208a2fb3512357e19db16d82073a7b7f74348418dc027d05e5b3f5fd7f3bbf4c"}
Oct 30 19:48:32.370396 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:32.370398    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"
Oct 30 19:48:32.370651 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:32.370419    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208a2fb3512357e19db16d82073a7b7f74348418dc027d05e5b3f5fd7f3bbf4c"
Oct 30 19:48:36.001371 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.001334    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"]
Oct 30 19:48:36.001897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.001848    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5eca6708-9134-4cde-ad50-7129cac56ea4" containerName="revoke-keys"
Oct 30 19:48:36.001897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.001863    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eca6708-9134-4cde-ad50-7129cac56ea4" containerName="revoke-keys"
Oct 30 19:48:36.001970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.001952    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eca6708-9134-4cde-ad50-7129cac56ea4" containerName="revoke-keys"
Oct 30 19:48:36.005654 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.005634    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.024305 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.024271    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"]
Oct 30 19:48:36.032762 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.032701    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:36.033430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.033400    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:48:36.098222 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.098178    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.098430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.098405    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.098487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.098451    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hx9x\" (UniqueName: \"kubernetes.io/projected/cf2b8778-84f0-4bf7-b2d2-c638cd709614-kube-api-access-8hx9x\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.199333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.199289    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.199333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.199336    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hx9x\" (UniqueName: \"kubernetes.io/projected/cf2b8778-84f0-4bf7-b2d2-c638cd709614-kube-api-access-8hx9x\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.199652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.199367    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.200144 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.200120    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.201866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.201840    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.219926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.219886    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hx9x\" (UniqueName: \"kubernetes.io/projected/cf2b8778-84f0-4bf7-b2d2-c638cd709614-kube-api-access-8hx9x\") pod \"maas-api-revoke-sub-aec88c6f-74xjp\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.321494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.321384    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:36.630574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:36.630541    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"]
Oct 30 19:48:36.631335 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:36.631306    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2b8778_84f0_4bf7_b2d2_c638cd709614.slice/crio-b3b256d2b9315e9536efd3aab64b7316e115157268c69fd88bfedcd8ed888f96 WatchSource:0}: Error finding container b3b256d2b9315e9536efd3aab64b7316e115157268c69fd88bfedcd8ed888f96: Status 404 returned error can't find the container with id b3b256d2b9315e9536efd3aab64b7316e115157268c69fd88bfedcd8ed888f96
Oct 30 19:48:37.414672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:37.414632    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp" event={"ID":"cf2b8778-84f0-4bf7-b2d2-c638cd709614","Type":"ContainerStarted","Data":"ccebcf7a0f27875310bb0c88ed8c63c18b6b92037c9db38effea2a09633b2868"}
Oct 30 19:48:37.415178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:37.414704    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp" event={"ID":"cf2b8778-84f0-4bf7-b2d2-c638cd709614","Type":"ContainerStarted","Data":"b3b256d2b9315e9536efd3aab64b7316e115157268c69fd88bfedcd8ed888f96"}
Oct 30 19:48:37.456938 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:37.456878    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp" podStartSLOduration=2.456863191 podStartE2EDuration="2.456863191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:37.456395646 +0000 UTC m=+841.957099370" watchObservedRunningTime="2026-10-30 19:48:37.456863191 +0000 UTC m=+841.957566915"
Oct 30 19:48:38.427762 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:38.427720    4519 generic.go:356] "Generic (PLEG): container finished" podID="cf2b8778-84f0-4bf7-b2d2-c638cd709614" containerID="ccebcf7a0f27875310bb0c88ed8c63c18b6b92037c9db38effea2a09633b2868" exitCode=0
Oct 30 19:48:38.428260 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:38.427819    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp" event={"ID":"cf2b8778-84f0-4bf7-b2d2-c638cd709614","Type":"ContainerDied","Data":"ccebcf7a0f27875310bb0c88ed8c63c18b6b92037c9db38effea2a09633b2868"}
Oct 30 19:48:39.621746 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.621714    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:39.740369 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.740267    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2b8778-84f0-4bf7-b2d2-c638cd709614-kube-api-access-8hx9x\" (UniqueName: \"kubernetes.io/projected/cf2b8778-84f0-4bf7-b2d2-c638cd709614-kube-api-access-8hx9x\") pod \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") "
Oct 30 19:48:39.740571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.740400    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-xks-ca\") pod \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") "
Oct 30 19:48:39.740571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.740558    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-openshift-ca\") pod \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\" (UID: \"cf2b8778-84f0-4bf7-b2d2-c638cd709614\") "
Oct 30 19:48:39.740939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.740903    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-openshift-ca" pod "cf2b8778-84f0-4bf7-b2d2-c638cd709614" (UID: "cf2b8778-84f0-4bf7-b2d2-c638cd709614"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:39.742513 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.742480    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2b8778-84f0-4bf7-b2d2-c638cd709614-kube-api-access-8hx9x" pod "cf2b8778-84f0-4bf7-b2d2-c638cd709614" (UID: "cf2b8778-84f0-4bf7-b2d2-c638cd709614"). InnerVolumeSpecName "kube-api-access-8hx9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:39.742618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.742519    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-xks-ca" pod "cf2b8778-84f0-4bf7-b2d2-c638cd709614" (UID: "cf2b8778-84f0-4bf7-b2d2-c638cd709614"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:39.842322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.842285    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:39.842322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.842318    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2b8778-84f0-4bf7-b2d2-c638cd709614-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:39.842322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:39.842331    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hx9x\" (UniqueName: \"kubernetes.io/projected/cf2b8778-84f0-4bf7-b2d2-c638cd709614-kube-api-access-8hx9x\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:40.441717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:40.441684    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"
Oct 30 19:48:40.441717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:40.441711    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp" event={"ID":"cf2b8778-84f0-4bf7-b2d2-c638cd709614","Type":"ContainerDied","Data":"b3b256d2b9315e9536efd3aab64b7316e115157268c69fd88bfedcd8ed888f96"}
Oct 30 19:48:40.441930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:40.441789    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b256d2b9315e9536efd3aab64b7316e115157268c69fd88bfedcd8ed888f96"
Oct 30 19:48:42.611795 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.611698    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"]
Oct 30 19:48:42.612377 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.612353    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cf2b8778-84f0-4bf7-b2d2-c638cd709614" containerName="revoke-keys"
Oct 30 19:48:42.612487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.612379    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2b8778-84f0-4bf7-b2d2-c638cd709614" containerName="revoke-keys"
Oct 30 19:48:42.612557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.612527    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2b8778-84f0-4bf7-b2d2-c638cd709614" containerName="revoke-keys"
Oct 30 19:48:42.616797 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.616772    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.631930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.631900    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8a3426-openshift-default-dockercfg-4xlnw\""
Oct 30 19:48:42.648707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.648664    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"]
Oct 30 19:48:42.674232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674182    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674239    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674272    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674377    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674433    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674459    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674538    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674586    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x744m\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-kube-api-access-x744m\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.674835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.674807    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776300    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776361    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776487    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776544    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776570    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x744m\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-kube-api-access-x744m\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776630    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776714    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776742    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776787    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.776853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.776809    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.777163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.777089    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.777241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.777164    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.777241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.777196    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.777241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.777230    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.779210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.779187    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.779325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.779224    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.796410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.796378    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.796580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.796456    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x744m\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-kube-api-access-x744m\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:42.972632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:42.972591    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:43.292897 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:43.292865    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e26568_997a_4e1b_a9c3_ec1a57085e7f.slice/crio-a38867aee166a3547c458d28f9b2258bae6cc52be559677b248d6f04affb337f WatchSource:0}: Error finding container a38867aee166a3547c458d28f9b2258bae6cc52be559677b248d6f04affb337f: Status 404 returned error can't find the container with id a38867aee166a3547c458d28f9b2258bae6cc52be559677b248d6f04affb337f
Oct 30 19:48:43.293339 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.293313    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"]
Oct 30 19:48:43.295149 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.295117    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:43.295237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.295184    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:43.295237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.295210    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:43.464510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.464456    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" event={"ID":"64e26568-997a-4e1b-a9c3-ec1a57085e7f","Type":"ContainerStarted","Data":"f7d5e8060e4fc9ca0dfa563fe49fccbce188bfe06e212fdce8f9338004c21d5f"}
Oct 30 19:48:43.464696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.464563    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" event={"ID":"64e26568-997a-4e1b-a9c3-ec1a57085e7f","Type":"ContainerStarted","Data":"a38867aee166a3547c458d28f9b2258bae6cc52be559677b248d6f04affb337f"}
Oct 30 19:48:43.511943 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.511871    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podStartSLOduration=1.511848219 podStartE2EDuration="1.511848219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:43.50889474 +0000 UTC m=+848.009598465" watchObservedRunningTime="2026-10-30 19:48:43.511848219 +0000 UTC m=+848.012551945"
Oct 30 19:48:43.972726 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:43.972675    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:44.979422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:44.979383    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:48:44.979951 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:44.979475    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:48:45.974492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:45.974265    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:48:45.974492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:45.974366    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:48:46.513820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.513776    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"]
Oct 30 19:48:46.517974 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.517952    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.543183 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.543142    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"]
Oct 30 19:48:46.582516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.571275    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:48:46.582516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.572147    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:46.620224 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.620185    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.620427 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.620250    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.620537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.620505    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwggz\" (UniqueName: \"kubernetes.io/projected/66d317c0-ef23-46f6-908e-9005eef1dbe9-kube-api-access-mwggz\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.722121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.722084    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwggz\" (UniqueName: \"kubernetes.io/projected/66d317c0-ef23-46f6-908e-9005eef1dbe9-kube-api-access-mwggz\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.722313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.722154    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.722313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.722195    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.722915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.722886    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.724761 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.724734    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.742939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.742869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwggz\" (UniqueName: \"kubernetes.io/projected/66d317c0-ef23-46f6-908e-9005eef1dbe9-kube-api-access-mwggz\") pod \"maas-api-revoke-sub-b8de6122-wk8fm\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.832887 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.832795    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:46.988356 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.983939    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:48:46.988356 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:46.984003    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:48:47.183235 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:47.183194    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d317c0_ef23_46f6_908e_9005eef1dbe9.slice/crio-afb7d965370d2290874b8ed5d342164627cb55b0ca395037a24aa1b92ee87058 WatchSource:0}: Error finding container afb7d965370d2290874b8ed5d342164627cb55b0ca395037a24aa1b92ee87058: Status 404 returned error can't find the container with id afb7d965370d2290874b8ed5d342164627cb55b0ca395037a24aa1b92ee87058
Oct 30 19:48:47.184724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:47.184440    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"]
Oct 30 19:48:47.515778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:47.515730    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm" event={"ID":"66d317c0-ef23-46f6-908e-9005eef1dbe9","Type":"ContainerStarted","Data":"1b980bf8e2e72064b594d97901450caa42a3e8a7b26e944eef2543acada3f9f7"}
Oct 30 19:48:47.516195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:47.515802    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm" event={"ID":"66d317c0-ef23-46f6-908e-9005eef1dbe9","Type":"ContainerStarted","Data":"afb7d965370d2290874b8ed5d342164627cb55b0ca395037a24aa1b92ee87058"}
Oct 30 19:48:47.979490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:47.976761    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:48:47.979490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:47.976823    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:48:48.523910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:48.523872    4519 generic.go:356] "Generic (PLEG): container finished" podID="66d317c0-ef23-46f6-908e-9005eef1dbe9" containerID="1b980bf8e2e72064b594d97901450caa42a3e8a7b26e944eef2543acada3f9f7" exitCode=0
Oct 30 19:48:48.524326 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:48.523963    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm" event={"ID":"66d317c0-ef23-46f6-908e-9005eef1dbe9","Type":"ContainerDied","Data":"1b980bf8e2e72064b594d97901450caa42a3e8a7b26e944eef2543acada3f9f7"}
Oct 30 19:48:48.977489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:48.976889    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:48:48.977489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:48.976953    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:48:49.399829 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.399728    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"]
Oct 30 19:48:49.403823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.403795    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.424514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.424446    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"]
Oct 30 19:48:49.459874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.459829    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqwd\" (UniqueName: \"kubernetes.io/projected/a9ea48e4-a64f-484d-acec-e5b5272b0801-kube-api-access-ptqwd\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.460102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.460008    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.460102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.460070    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.561882 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.561826    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.562309 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.562008    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqwd\" (UniqueName: \"kubernetes.io/projected/a9ea48e4-a64f-484d-acec-e5b5272b0801-kube-api-access-ptqwd\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.562309 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.562094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.563017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.562976    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.570483 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.570427    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.590682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.590642    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqwd\" (UniqueName: \"kubernetes.io/projected/a9ea48e4-a64f-484d-acec-e5b5272b0801-kube-api-access-ptqwd\") pod \"maas-api-revoke-sub-40cacd41-ltbvf\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.707250 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.707219    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:49.717505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.717430    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:49.765422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.764676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d317c0-ef23-46f6-908e-9005eef1dbe9-kube-api-access-mwggz\" (UniqueName: \"kubernetes.io/projected/66d317c0-ef23-46f6-908e-9005eef1dbe9-kube-api-access-mwggz\") pod \"66d317c0-ef23-46f6-908e-9005eef1dbe9\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") "
Oct 30 19:48:49.765422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.764746    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-xks-ca\") pod \"66d317c0-ef23-46f6-908e-9005eef1dbe9\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") "
Oct 30 19:48:49.765422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.764813    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-openshift-ca\") pod \"66d317c0-ef23-46f6-908e-9005eef1dbe9\" (UID: \"66d317c0-ef23-46f6-908e-9005eef1dbe9\") "
Oct 30 19:48:49.766283 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.765933    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-openshift-ca" pod "66d317c0-ef23-46f6-908e-9005eef1dbe9" (UID: "66d317c0-ef23-46f6-908e-9005eef1dbe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:49.799460 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.799394    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d317c0-ef23-46f6-908e-9005eef1dbe9-kube-api-access-mwggz" pod "66d317c0-ef23-46f6-908e-9005eef1dbe9" (UID: "66d317c0-ef23-46f6-908e-9005eef1dbe9"). InnerVolumeSpecName "kube-api-access-mwggz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.802131 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.802094    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-xks-ca" pod "66d317c0-ef23-46f6-908e-9005eef1dbe9" (UID: "66d317c0-ef23-46f6-908e-9005eef1dbe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:49.866450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.866412    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwggz\" (UniqueName: \"kubernetes.io/projected/66d317c0-ef23-46f6-908e-9005eef1dbe9-kube-api-access-mwggz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:49.866450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.866447    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:49.866450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.866460    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d317c0-ef23-46f6-908e-9005eef1dbe9-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:49.974056 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.973951    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:48:49.974056 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:49.974031    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:48:50.038327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.038278    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"]
Oct 30 19:48:50.038776 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:50.038748    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ea48e4_a64f_484d_acec_e5b5272b0801.slice/crio-71c2ce6c2a2067630c480d2cb62f624cfc20a3927bb50ac8168da1aa5ed357f2 WatchSource:0}: Error finding container 71c2ce6c2a2067630c480d2cb62f624cfc20a3927bb50ac8168da1aa5ed357f2: Status 404 returned error can't find the container with id 71c2ce6c2a2067630c480d2cb62f624cfc20a3927bb50ac8168da1aa5ed357f2
Oct 30 19:48:50.538935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.538888    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm" event={"ID":"66d317c0-ef23-46f6-908e-9005eef1dbe9","Type":"ContainerDied","Data":"afb7d965370d2290874b8ed5d342164627cb55b0ca395037a24aa1b92ee87058"}
Oct 30 19:48:50.539103 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.538965    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afb7d965370d2290874b8ed5d342164627cb55b0ca395037a24aa1b92ee87058"
Oct 30 19:48:50.539103 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.538904    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"
Oct 30 19:48:50.540581 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.540553    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf" event={"ID":"a9ea48e4-a64f-484d-acec-e5b5272b0801","Type":"ContainerStarted","Data":"675bebcbad23673af6c52df4e4757d348e0317d6561a4f50d83cec8d47dee6ee"}
Oct 30 19:48:50.540763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.540742    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf" event={"ID":"a9ea48e4-a64f-484d-acec-e5b5272b0801","Type":"ContainerStarted","Data":"71c2ce6c2a2067630c480d2cb62f624cfc20a3927bb50ac8168da1aa5ed357f2"}
Oct 30 19:48:50.584240 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.584169    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf" podStartSLOduration=1.584150203 podStartE2EDuration="1.584150203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:50.579276537 +0000 UTC m=+855.079980262" watchObservedRunningTime="2026-10-30 19:48:50.584150203 +0000 UTC m=+855.084853934"
Oct 30 19:48:50.978850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.978816    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:50.979261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.979238    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:50.980106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:50.980089    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 19:48:51.548650 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:51.548610    4519 generic.go:356] "Generic (PLEG): container finished" podID="a9ea48e4-a64f-484d-acec-e5b5272b0801" containerID="675bebcbad23673af6c52df4e4757d348e0317d6561a4f50d83cec8d47dee6ee" exitCode=0
Oct 30 19:48:51.548872 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:51.548701    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf" event={"ID":"a9ea48e4-a64f-484d-acec-e5b5272b0801","Type":"ContainerDied","Data":"675bebcbad23673af6c52df4e4757d348e0317d6561a4f50d83cec8d47dee6ee"}
Oct 30 19:48:52.880502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:52.880401    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:53.013502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.013427    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-openshift-ca\") pod \"a9ea48e4-a64f-484d-acec-e5b5272b0801\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") "
Oct 30 19:48:53.013712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.013572    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ea48e4-a64f-484d-acec-e5b5272b0801-kube-api-access-ptqwd\" (UniqueName: \"kubernetes.io/projected/a9ea48e4-a64f-484d-acec-e5b5272b0801-kube-api-access-ptqwd\") pod \"a9ea48e4-a64f-484d-acec-e5b5272b0801\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") "
Oct 30 19:48:53.013712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.013678    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-xks-ca\") pod \"a9ea48e4-a64f-484d-acec-e5b5272b0801\" (UID: \"a9ea48e4-a64f-484d-acec-e5b5272b0801\") "
Oct 30 19:48:53.013943 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.013910    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-openshift-ca" pod "a9ea48e4-a64f-484d-acec-e5b5272b0801" (UID: "a9ea48e4-a64f-484d-acec-e5b5272b0801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:53.014077 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.014050    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:53.015834 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.015806    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ea48e4-a64f-484d-acec-e5b5272b0801-kube-api-access-ptqwd" pod "a9ea48e4-a64f-484d-acec-e5b5272b0801" (UID: "a9ea48e4-a64f-484d-acec-e5b5272b0801"). InnerVolumeSpecName "kube-api-access-ptqwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:53.015941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.015892    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-xks-ca" pod "a9ea48e4-a64f-484d-acec-e5b5272b0801" (UID: "a9ea48e4-a64f-484d-acec-e5b5272b0801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:53.115642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.115543    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqwd\" (UniqueName: \"kubernetes.io/projected/a9ea48e4-a64f-484d-acec-e5b5272b0801-kube-api-access-ptqwd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:53.115642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.115580    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ea48e4-a64f-484d-acec-e5b5272b0801-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:48:53.572807 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.572751    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf" event={"ID":"a9ea48e4-a64f-484d-acec-e5b5272b0801","Type":"ContainerDied","Data":"71c2ce6c2a2067630c480d2cb62f624cfc20a3927bb50ac8168da1aa5ed357f2"}
Oct 30 19:48:53.572987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.572832    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71c2ce6c2a2067630c480d2cb62f624cfc20a3927bb50ac8168da1aa5ed357f2"
Oct 30 19:48:53.572987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:53.572765    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"
Oct 30 19:48:56.731483 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.731428    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"]
Oct 30 19:48:56.732227 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.732205    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a9ea48e4-a64f-484d-acec-e5b5272b0801" containerName="revoke-keys"
Oct 30 19:48:56.732227 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.732228    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ea48e4-a64f-484d-acec-e5b5272b0801" containerName="revoke-keys"
Oct 30 19:48:56.732338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.732246    4519 cpu_manager.go:414] "containerMap: removing container" podUID="66d317c0-ef23-46f6-908e-9005eef1dbe9" containerName="revoke-keys"
Oct 30 19:48:56.732338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.732251    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d317c0-ef23-46f6-908e-9005eef1dbe9" containerName="revoke-keys"
Oct 30 19:48:56.732338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.732321    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d317c0-ef23-46f6-908e-9005eef1dbe9" containerName="revoke-keys"
Oct 30 19:48:56.732338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.732334    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ea48e4-a64f-484d-acec-e5b5272b0801" containerName="revoke-keys"
Oct 30 19:48:56.739242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.739213    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.768164 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.767753    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"]
Oct 30 19:48:56.842972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.842927    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"]
Oct 30 19:48:56.849454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.849418    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:56.859977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.859944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b791e0b1-579c-4491-8226-418ad0459043-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860104 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.859990    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860104 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.860020    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.860124    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlxl\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-kube-api-access-ljlxl\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.860250    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.860295    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b791e0b1-579c-4491-8226-418ad0459043-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.860325    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860416 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.860401    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.860526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.860512    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.866350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.866323    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:56.866524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.866400    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:48:56.870874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.870842    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"]
Oct 30 19:48:56.961956 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.961915    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.961956 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.961967    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlxl\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-kube-api-access-ljlxl\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962022    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:56.962288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962085    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b791e0b1-579c-4491-8226-418ad0459043-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962118    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:56.962288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962153    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962188    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962254    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7xwr\" (UniqueName: \"kubernetes.io/projected/88ba08c6-7c2d-45a2-9403-12258a8f98ff-kube-api-access-p7xwr\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:56.962640 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962302    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962640 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962391    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b791e0b1-579c-4491-8226-418ad0459043-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962640 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962420    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962640 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962525    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.962868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.962828    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.963304 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.963279    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b791e0b1-579c-4491-8226-418ad0459043-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.963655 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.963631    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.963773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.963752    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.965475 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.965439    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.965812 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.965786    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b791e0b1-579c-4491-8226-418ad0459043-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.983249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.983168    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlxl\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-kube-api-access-ljlxl\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:56.983938 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:56.983902    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:57.055496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.055433    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:57.064102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.064058    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:57.064272 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.064129    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:57.064272 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.064211    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7xwr\" (UniqueName: \"kubernetes.io/projected/88ba08c6-7c2d-45a2-9403-12258a8f98ff-kube-api-access-p7xwr\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:57.065034 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.064976    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:57.067333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.067306    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:57.087532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.087454    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7xwr\" (UniqueName: \"kubernetes.io/projected/88ba08c6-7c2d-45a2-9403-12258a8f98ff-kube-api-access-p7xwr\") pod \"maas-api-revoke-sub-0fc544d5-s695n\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:57.164692 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.164647    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:48:57.387646 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:57.387605    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb791e0b1_579c_4491_8226_418ad0459043.slice/crio-1998c207744d19bdbc666bba06451d106e51a83469c76384f6f3039d7e42a33d WatchSource:0}: Error finding container 1998c207744d19bdbc666bba06451d106e51a83469c76384f6f3039d7e42a33d: Status 404 returned error can't find the container with id 1998c207744d19bdbc666bba06451d106e51a83469c76384f6f3039d7e42a33d
Oct 30 19:48:57.388226 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.388197    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"]
Oct 30 19:48:57.390531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.390449    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:57.390624 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.390577    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:57.390667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.390619    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:48:57.491365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.491277    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"]
Oct 30 19:48:57.491887 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:48:57.491842    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ba08c6_7c2d_45a2_9403_12258a8f98ff.slice/crio-bf68cb0109b4b3d29aee98fb28ddff0fd459587882fc64eae79c9a86e368c24c WatchSource:0}: Error finding container bf68cb0109b4b3d29aee98fb28ddff0fd459587882fc64eae79c9a86e368c24c: Status 404 returned error can't find the container with id bf68cb0109b4b3d29aee98fb28ddff0fd459587882fc64eae79c9a86e368c24c
Oct 30 19:48:57.604173 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.604133    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" event={"ID":"b791e0b1-579c-4491-8226-418ad0459043","Type":"ContainerStarted","Data":"d55a6de033dde18f00607a4d8e7ba77e9f7cdc2ac8d8e475efb17e9a9c6e0115"}
Oct 30 19:48:57.604387 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.604205    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" event={"ID":"b791e0b1-579c-4491-8226-418ad0459043","Type":"ContainerStarted","Data":"1998c207744d19bdbc666bba06451d106e51a83469c76384f6f3039d7e42a33d"}
Oct 30 19:48:57.613761 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.613720    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n" event={"ID":"88ba08c6-7c2d-45a2-9403-12258a8f98ff","Type":"ContainerStarted","Data":"bf68cb0109b4b3d29aee98fb28ddff0fd459587882fc64eae79c9a86e368c24c"}
Oct 30 19:48:57.659065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:57.659002    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podStartSLOduration=1.658986195 podStartE2EDuration="1.658986195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:57.654595108 +0000 UTC m=+862.155298832" watchObservedRunningTime="2026-10-30 19:48:57.658986195 +0000 UTC m=+862.159689918"
Oct 30 19:48:58.055689 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:58.055585    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:48:58.057404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:58.057371    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:48:58.057619 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:58.057435    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:48:58.613773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:58.613735    4519 generic.go:356] "Generic (PLEG): container finished" podID="88ba08c6-7c2d-45a2-9403-12258a8f98ff" containerID="ea7e8f2b4d915d164a91d77a1d9d9368999f3fb4b50600c12ca07810758a262c" exitCode=0
Oct 30 19:48:58.614027 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:58.613820    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n" event={"ID":"88ba08c6-7c2d-45a2-9403-12258a8f98ff","Type":"ContainerDied","Data":"ea7e8f2b4d915d164a91d77a1d9d9368999f3fb4b50600c12ca07810758a262c"}
Oct 30 19:48:59.055980 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:59.055938    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:48:59.057391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:59.057354    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:48:59.838522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:48:59.838486    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:49:00.008122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.008084    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-xks-ca\") pod \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") "
Oct 30 19:49:00.008306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.008189    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-openshift-ca\") pod \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") "
Oct 30 19:49:00.008306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.008227    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ba08c6-7c2d-45a2-9403-12258a8f98ff-kube-api-access-p7xwr\" (UniqueName: \"kubernetes.io/projected/88ba08c6-7c2d-45a2-9403-12258a8f98ff-kube-api-access-p7xwr\") pod \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\" (UID: \"88ba08c6-7c2d-45a2-9403-12258a8f98ff\") "
Oct 30 19:49:00.008622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.008600    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-openshift-ca" pod "88ba08c6-7c2d-45a2-9403-12258a8f98ff" (UID: "88ba08c6-7c2d-45a2-9403-12258a8f98ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:00.010410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.010371    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-xks-ca" pod "88ba08c6-7c2d-45a2-9403-12258a8f98ff" (UID: "88ba08c6-7c2d-45a2-9403-12258a8f98ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:00.010582 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.010478    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ba08c6-7c2d-45a2-9403-12258a8f98ff-kube-api-access-p7xwr" pod "88ba08c6-7c2d-45a2-9403-12258a8f98ff" (UID: "88ba08c6-7c2d-45a2-9403-12258a8f98ff"). InnerVolumeSpecName "kube-api-access-p7xwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:00.058002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.057957    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:49:00.058653 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.058619    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:49:00.109399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.109358    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:00.109399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.109395    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7xwr\" (UniqueName: \"kubernetes.io/projected/88ba08c6-7c2d-45a2-9403-12258a8f98ff-kube-api-access-p7xwr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:00.109399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.109411    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ba08c6-7c2d-45a2-9403-12258a8f98ff-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:00.627961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.627923    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"
Oct 30 19:49:00.627961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.627955    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n" event={"ID":"88ba08c6-7c2d-45a2-9403-12258a8f98ff","Type":"ContainerDied","Data":"bf68cb0109b4b3d29aee98fb28ddff0fd459587882fc64eae79c9a86e368c24c"}
Oct 30 19:49:00.628185 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:00.628009    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf68cb0109b4b3d29aee98fb28ddff0fd459587882fc64eae79c9a86e368c24c"
Oct 30 19:49:01.064934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:01.064893    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:49:01.065579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:01.065542    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:49:02.067960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:02.067920    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:49:02.068596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:02.068561    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:49:02.290296 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:49:02.290250    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:49:03.061494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:03.059899    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:49:03.061494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:03.059980    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:49:04.056023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:04.055980    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:49:04.056484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:04.056060    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:49:05.291225 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.291179    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:49:05.291944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.291920    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:49:05.293544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.293512    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 19:49:05.766772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.766694    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"]
Oct 30 19:49:05.767447 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.767428    4519 cpu_manager.go:414] "containerMap: removing container" podUID="88ba08c6-7c2d-45a2-9403-12258a8f98ff" containerName="revoke-keys"
Oct 30 19:49:05.767563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.767449    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ba08c6-7c2d-45a2-9403-12258a8f98ff" containerName="revoke-keys"
Oct 30 19:49:05.767623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.767585    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ba08c6-7c2d-45a2-9403-12258a8f98ff" containerName="revoke-keys"
Oct 30 19:49:05.771828 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.771797    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:05.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.789695    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:05.789926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.789785    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:49:05.793895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.793856    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"]
Oct 30 19:49:05.920610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.920563    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znt8\" (UniqueName: \"kubernetes.io/projected/27a15706-2d2b-402f-bbe7-860bf13f8264-kube-api-access-6znt8\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:05.920798 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.920682    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:05.920840 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:05.920795    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.021869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.021782    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znt8\" (UniqueName: \"kubernetes.io/projected/27a15706-2d2b-402f-bbe7-860bf13f8264-kube-api-access-6znt8\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.021869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.021837    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.022071 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.021895    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.022744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.022713    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.024533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.024504    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.041900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.041870    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znt8\" (UniqueName: \"kubernetes.io/projected/27a15706-2d2b-402f-bbe7-860bf13f8264-kube-api-access-6znt8\") pod \"maas-api-revoke-sub-09a0c2c0-kqw9p\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.089586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.089530    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:06.402966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.402855    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"]
Oct 30 19:49:06.404382 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:06.404347    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a15706_2d2b_402f_bbe7_860bf13f8264.slice/crio-c9eccbe5179df5a16a75f21f1ce0f2ff5d7f4b47ad48c9ffab80209d64632f62 WatchSource:0}: Error finding container c9eccbe5179df5a16a75f21f1ce0f2ff5d7f4b47ad48c9ffab80209d64632f62: Status 404 returned error can't find the container with id c9eccbe5179df5a16a75f21f1ce0f2ff5d7f4b47ad48c9ffab80209d64632f62
Oct 30 19:49:06.706109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.706074    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p" event={"ID":"27a15706-2d2b-402f-bbe7-860bf13f8264","Type":"ContainerStarted","Data":"c9eccbe5179df5a16a75f21f1ce0f2ff5d7f4b47ad48c9ffab80209d64632f62"}
Oct 30 19:49:06.835790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.835749    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"]
Oct 30 19:49:06.840610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.840582    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:06.868853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.868814    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"]
Oct 30 19:49:06.874662 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.874626    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"]
Oct 30 19:49:06.878620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.878594    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:06.905454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.905422    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"]
Oct 30 19:49:06.934819 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.934778    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24hr\" (UniqueName: \"kubernetes.io/projected/8eabf850-bde4-4d95-8740-2936222afb33-kube-api-access-z24hr\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:06.935039 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.934868    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:06.935039 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:06.934943    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.036255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.036213    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.036436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.036328    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.036436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.036385    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knssr\" (UniqueName: \"kubernetes.io/projected/abd6f340-f243-4dc1-b425-afa89d505e69-kube-api-access-knssr\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.036569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.036500    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.036614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.036567    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24hr\" (UniqueName: \"kubernetes.io/projected/8eabf850-bde4-4d95-8740-2936222afb33-kube-api-access-z24hr\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.036614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.036607    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.037101 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.037076    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.039068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.039039    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.059897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.059866    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24hr\" (UniqueName: \"kubernetes.io/projected/8eabf850-bde4-4d95-8740-2936222afb33-kube-api-access-z24hr\") pod \"maas-api-revoke-sub-1d61bf1d-tvbh9\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.138018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.137981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.138198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.138043    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knssr\" (UniqueName: \"kubernetes.io/projected/abd6f340-f243-4dc1-b425-afa89d505e69-kube-api-access-knssr\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.138198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.138103    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.138733 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.138712    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.140504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.140483    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.157306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.157269    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:07.159143 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.159114    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knssr\" (UniqueName: \"kubernetes.io/projected/abd6f340-f243-4dc1-b425-afa89d505e69-kube-api-access-knssr\") pod \"maas-api-revoke-sub-d42b0ec6-kjd2l\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.202536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.202441    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:07.237752 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.232684    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"]
Oct 30 19:49:07.239827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.239798    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.257677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.257623    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"]
Oct 30 19:49:07.340719 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.340684    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.340914 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.340737    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.340914 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.340843    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qck4b\" (UniqueName: \"kubernetes.io/projected/77e6279a-f362-4067-bcdb-59b369f04c71-kube-api-access-qck4b\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.442414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.442375    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.442949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.442434    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.442949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.442459    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qck4b\" (UniqueName: \"kubernetes.io/projected/77e6279a-f362-4067-bcdb-59b369f04c71-kube-api-access-qck4b\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.443284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.443258    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.445091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.445071    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.463846 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.463764    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qck4b\" (UniqueName: \"kubernetes.io/projected/77e6279a-f362-4067-bcdb-59b369f04c71-kube-api-access-qck4b\") pod \"maas-api-revoke-sub-66cece07-2scnn\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.470794 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:07.470755    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eabf850_bde4_4d95_8740_2936222afb33.slice/crio-2f34191ecf252d1f1061891e3bfdb18a0bf6b131641558c800b4c36330afd2a9 WatchSource:0}: Error finding container 2f34191ecf252d1f1061891e3bfdb18a0bf6b131641558c800b4c36330afd2a9: Status 404 returned error can't find the container with id 2f34191ecf252d1f1061891e3bfdb18a0bf6b131641558c800b4c36330afd2a9
Oct 30 19:49:07.472206 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.472177    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"]
Oct 30 19:49:07.545383 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.545352    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"]
Oct 30 19:49:07.545573 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:07.545549    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd6f340_f243_4dc1_b425_afa89d505e69.slice/crio-dea0232c8d5265bc3f8a3e222ba6b629d73ae6d293812b267f879828ef8ccb6f WatchSource:0}: Error finding container dea0232c8d5265bc3f8a3e222ba6b629d73ae6d293812b267f879828ef8ccb6f: Status 404 returned error can't find the container with id dea0232c8d5265bc3f8a3e222ba6b629d73ae6d293812b267f879828ef8ccb6f
Oct 30 19:49:07.561823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.561783    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:07.716963 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.716870    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l" event={"ID":"abd6f340-f243-4dc1-b425-afa89d505e69","Type":"ContainerStarted","Data":"dea0232c8d5265bc3f8a3e222ba6b629d73ae6d293812b267f879828ef8ccb6f"}
Oct 30 19:49:07.718673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.718643    4519 generic.go:356] "Generic (PLEG): container finished" podID="27a15706-2d2b-402f-bbe7-860bf13f8264" containerID="41104fbaae2c42b017bda3db037ff24b39efd9c2114fc63728e9da187e87db46" exitCode=0
Oct 30 19:49:07.718816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.718728    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p" event={"ID":"27a15706-2d2b-402f-bbe7-860bf13f8264","Type":"ContainerDied","Data":"41104fbaae2c42b017bda3db037ff24b39efd9c2114fc63728e9da187e87db46"}
Oct 30 19:49:07.720039 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.720014    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9" event={"ID":"8eabf850-bde4-4d95-8740-2936222afb33","Type":"ContainerStarted","Data":"2f34191ecf252d1f1061891e3bfdb18a0bf6b131641558c800b4c36330afd2a9"}
Oct 30 19:49:07.876483 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:07.876291    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"]
Oct 30 19:49:08.729250 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.729148    4519 generic.go:356] "Generic (PLEG): container finished" podID="8eabf850-bde4-4d95-8740-2936222afb33" containerID="0e7679040cc801bc097d34b8a8bf011d0918b68b05fb00a03693fc2f36e8b639" exitCode=0
Oct 30 19:49:08.729250 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.729232    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9" event={"ID":"8eabf850-bde4-4d95-8740-2936222afb33","Type":"ContainerDied","Data":"0e7679040cc801bc097d34b8a8bf011d0918b68b05fb00a03693fc2f36e8b639"}
Oct 30 19:49:08.731084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.731039    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn" event={"ID":"77e6279a-f362-4067-bcdb-59b369f04c71","Type":"ContainerStarted","Data":"beda752664e5aed32b301add03c37ed2c786919e02ed2fd0c9511736e439ed15"}
Oct 30 19:49:08.731229 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.731110    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn" event={"ID":"77e6279a-f362-4067-bcdb-59b369f04c71","Type":"ContainerStarted","Data":"8d8194ea14daa4e7758767a66df1a2b809b43638c2cdc3f056e635453005abcd"}
Oct 30 19:49:08.732924 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.732895    4519 generic.go:356] "Generic (PLEG): container finished" podID="abd6f340-f243-4dc1-b425-afa89d505e69" containerID="cf05c25ba45645c7114ca1f46dfb350da7c3647d6200672d24eb7b359ac8f3a4" exitCode=0
Oct 30 19:49:08.733068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.732932    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l" event={"ID":"abd6f340-f243-4dc1-b425-afa89d505e69","Type":"ContainerDied","Data":"cf05c25ba45645c7114ca1f46dfb350da7c3647d6200672d24eb7b359ac8f3a4"}
Oct 30 19:49:08.821671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.821604    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn" podStartSLOduration=1.821588134 podStartE2EDuration="1.821588134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:08.820064482 +0000 UTC m=+873.320768210" watchObservedRunningTime="2026-10-30 19:49:08.821588134 +0000 UTC m=+873.322291896"
Oct 30 19:49:08.896332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:08.896293    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:09.066270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.066169    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a15706-2d2b-402f-bbe7-860bf13f8264-kube-api-access-6znt8\" (UniqueName: \"kubernetes.io/projected/27a15706-2d2b-402f-bbe7-860bf13f8264-kube-api-access-6znt8\") pod \"27a15706-2d2b-402f-bbe7-860bf13f8264\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") "
Oct 30 19:49:09.066270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.066251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-openshift-ca\") pod \"27a15706-2d2b-402f-bbe7-860bf13f8264\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") "
Oct 30 19:49:09.066531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.066313    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-xks-ca\") pod \"27a15706-2d2b-402f-bbe7-860bf13f8264\" (UID: \"27a15706-2d2b-402f-bbe7-860bf13f8264\") "
Oct 30 19:49:09.066799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.066758    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-openshift-ca" pod "27a15706-2d2b-402f-bbe7-860bf13f8264" (UID: "27a15706-2d2b-402f-bbe7-860bf13f8264"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:09.068745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.068714    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-xks-ca" pod "27a15706-2d2b-402f-bbe7-860bf13f8264" (UID: "27a15706-2d2b-402f-bbe7-860bf13f8264"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:09.068843 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.068719    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a15706-2d2b-402f-bbe7-860bf13f8264-kube-api-access-6znt8" pod "27a15706-2d2b-402f-bbe7-860bf13f8264" (UID: "27a15706-2d2b-402f-bbe7-860bf13f8264"). InnerVolumeSpecName "kube-api-access-6znt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:09.168319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.168280    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znt8\" (UniqueName: \"kubernetes.io/projected/27a15706-2d2b-402f-bbe7-860bf13f8264-kube-api-access-6znt8\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:09.168319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.168316    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:09.168319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.168326    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27a15706-2d2b-402f-bbe7-860bf13f8264-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:09.585820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.585773    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"]
Oct 30 19:49:09.586425 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.586404    4519 cpu_manager.go:414] "containerMap: removing container" podUID="27a15706-2d2b-402f-bbe7-860bf13f8264" containerName="revoke-keys"
Oct 30 19:49:09.586557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.586427    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a15706-2d2b-402f-bbe7-860bf13f8264" containerName="revoke-keys"
Oct 30 19:49:09.586617 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.586584    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a15706-2d2b-402f-bbe7-860bf13f8264" containerName="revoke-keys"
Oct 30 19:49:09.590230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.590203    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.614364 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.614329    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"]
Oct 30 19:49:09.675451 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.675399    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dt8w\" (UniqueName: \"kubernetes.io/projected/9f171125-a5d1-4a3a-b6d5-8b56531ef647-kube-api-access-8dt8w\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.675706 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.675617    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.675706 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.675676    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.741122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.741093    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"
Oct 30 19:49:09.741122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.741118    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p" event={"ID":"27a15706-2d2b-402f-bbe7-860bf13f8264","Type":"ContainerDied","Data":"c9eccbe5179df5a16a75f21f1ce0f2ff5d7f4b47ad48c9ffab80209d64632f62"}
Oct 30 19:49:09.741699 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.741167    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9eccbe5179df5a16a75f21f1ce0f2ff5d7f4b47ad48c9ffab80209d64632f62"
Oct 30 19:49:09.743587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.743559    4519 generic.go:356] "Generic (PLEG): container finished" podID="77e6279a-f362-4067-bcdb-59b369f04c71" containerID="beda752664e5aed32b301add03c37ed2c786919e02ed2fd0c9511736e439ed15" exitCode=0
Oct 30 19:49:09.743829 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.743808    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn" event={"ID":"77e6279a-f362-4067-bcdb-59b369f04c71","Type":"ContainerDied","Data":"beda752664e5aed32b301add03c37ed2c786919e02ed2fd0c9511736e439ed15"}
Oct 30 19:49:09.778064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.776733    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dt8w\" (UniqueName: \"kubernetes.io/projected/9f171125-a5d1-4a3a-b6d5-8b56531ef647-kube-api-access-8dt8w\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.778064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.776833    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.778064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.776862    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.778064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.777931    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.783675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.783459    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.797947 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.797843    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dt8w\" (UniqueName: \"kubernetes.io/projected/9f171125-a5d1-4a3a-b6d5-8b56531ef647-kube-api-access-8dt8w\") pod \"maas-api-revoke-sub-19ade06a-mnfmm\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.906400 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.906361    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:09.925547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.925330    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:09.958789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:09.958666    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:10.081197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.081150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-xks-ca\") pod \"8eabf850-bde4-4d95-8740-2936222afb33\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") "
Oct 30 19:49:10.081413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.081224    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-xks-ca\") pod \"abd6f340-f243-4dc1-b425-afa89d505e69\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") "
Oct 30 19:49:10.081413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.081332    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-openshift-ca\") pod \"8eabf850-bde4-4d95-8740-2936222afb33\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") "
Oct 30 19:49:10.081413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.081375    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-openshift-ca\") pod \"abd6f340-f243-4dc1-b425-afa89d505e69\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") "
Oct 30 19:49:10.081413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.081395    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd6f340-f243-4dc1-b425-afa89d505e69-kube-api-access-knssr\" (UniqueName: \"kubernetes.io/projected/abd6f340-f243-4dc1-b425-afa89d505e69-kube-api-access-knssr\") pod \"abd6f340-f243-4dc1-b425-afa89d505e69\" (UID: \"abd6f340-f243-4dc1-b425-afa89d505e69\") "
Oct 30 19:49:10.081665 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.081423    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eabf850-bde4-4d95-8740-2936222afb33-kube-api-access-z24hr\" (UniqueName: \"kubernetes.io/projected/8eabf850-bde4-4d95-8740-2936222afb33-kube-api-access-z24hr\") pod \"8eabf850-bde4-4d95-8740-2936222afb33\" (UID: \"8eabf850-bde4-4d95-8740-2936222afb33\") "
Oct 30 19:49:10.082589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.082389    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-openshift-ca" pod "8eabf850-bde4-4d95-8740-2936222afb33" (UID: "8eabf850-bde4-4d95-8740-2936222afb33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:10.082589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.082505    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-openshift-ca" pod "abd6f340-f243-4dc1-b425-afa89d505e69" (UID: "abd6f340-f243-4dc1-b425-afa89d505e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:10.084749 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.084718    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eabf850-bde4-4d95-8740-2936222afb33-kube-api-access-z24hr" pod "8eabf850-bde4-4d95-8740-2936222afb33" (UID: "8eabf850-bde4-4d95-8740-2936222afb33"). InnerVolumeSpecName "kube-api-access-z24hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:10.084875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.084811    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-xks-ca" pod "abd6f340-f243-4dc1-b425-afa89d505e69" (UID: "abd6f340-f243-4dc1-b425-afa89d505e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:10.084875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.084806    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-xks-ca" pod "8eabf850-bde4-4d95-8740-2936222afb33" (UID: "8eabf850-bde4-4d95-8740-2936222afb33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:10.084983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.084924    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd6f340-f243-4dc1-b425-afa89d505e69-kube-api-access-knssr" pod "abd6f340-f243-4dc1-b425-afa89d505e69" (UID: "abd6f340-f243-4dc1-b425-afa89d505e69"). InnerVolumeSpecName "kube-api-access-knssr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:10.183000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.182956    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:10.183000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.182993    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:10.183000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.183003    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eabf850-bde4-4d95-8740-2936222afb33-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:10.183000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.183013    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6f340-f243-4dc1-b425-afa89d505e69-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:10.183299 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.183022    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knssr\" (UniqueName: \"kubernetes.io/projected/abd6f340-f243-4dc1-b425-afa89d505e69-kube-api-access-knssr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:10.183299 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.183031    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24hr\" (UniqueName: \"kubernetes.io/projected/8eabf850-bde4-4d95-8740-2936222afb33-kube-api-access-z24hr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:10.262842 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:10.262807    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f171125_a5d1_4a3a_b6d5_8b56531ef647.slice/crio-1032799ef248b9b63d824d11fa8a4f3b94904f225f163677cabbdf1e81e380ed WatchSource:0}: Error finding container 1032799ef248b9b63d824d11fa8a4f3b94904f225f163677cabbdf1e81e380ed: Status 404 returned error can't find the container with id 1032799ef248b9b63d824d11fa8a4f3b94904f225f163677cabbdf1e81e380ed
Oct 30 19:49:10.263232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.263206    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"]
Oct 30 19:49:10.808255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.808125    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9" event={"ID":"8eabf850-bde4-4d95-8740-2936222afb33","Type":"ContainerDied","Data":"2f34191ecf252d1f1061891e3bfdb18a0bf6b131641558c800b4c36330afd2a9"}
Oct 30 19:49:10.808895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.808855    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f34191ecf252d1f1061891e3bfdb18a0bf6b131641558c800b4c36330afd2a9"
Oct 30 19:49:10.809482 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.808999    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"
Oct 30 19:49:10.818333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.818291    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l" event={"ID":"abd6f340-f243-4dc1-b425-afa89d505e69","Type":"ContainerDied","Data":"dea0232c8d5265bc3f8a3e222ba6b629d73ae6d293812b267f879828ef8ccb6f"}
Oct 30 19:49:10.818666 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.818643    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea0232c8d5265bc3f8a3e222ba6b629d73ae6d293812b267f879828ef8ccb6f"
Oct 30 19:49:10.818929 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.818912    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"
Oct 30 19:49:10.849493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.843651    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm" event={"ID":"9f171125-a5d1-4a3a-b6d5-8b56531ef647","Type":"ContainerStarted","Data":"60633fb7f7fb846b2ff9ca8f26d0614c32691ad0701e0cce21590f9162262f40"}
Oct 30 19:49:10.887492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.865347    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm" event={"ID":"9f171125-a5d1-4a3a-b6d5-8b56531ef647","Type":"ContainerStarted","Data":"1032799ef248b9b63d824d11fa8a4f3b94904f225f163677cabbdf1e81e380ed"}
Oct 30 19:49:10.960491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:10.951401    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm" podStartSLOduration=1.951379861 podStartE2EDuration="1.951379861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:10.951237147 +0000 UTC m=+875.451940872" watchObservedRunningTime="2026-10-30 19:49:10.951379861 +0000 UTC m=+875.452083586"
Oct 30 19:49:11.593523 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.567732    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:11.767986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.767853    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"]
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768614    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8eabf850-bde4-4d95-8740-2936222afb33" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768651    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eabf850-bde4-4d95-8740-2936222afb33" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768673    4519 cpu_manager.go:414] "containerMap: removing container" podUID="77e6279a-f362-4067-bcdb-59b369f04c71" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768682    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e6279a-f362-4067-bcdb-59b369f04c71" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768708    4519 cpu_manager.go:414] "containerMap: removing container" podUID="abd6f340-f243-4dc1-b425-afa89d505e69" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768718    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd6f340-f243-4dc1-b425-afa89d505e69" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768830    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd6f340-f243-4dc1-b425-afa89d505e69" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768849    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eabf850-bde4-4d95-8740-2936222afb33" containerName="revoke-keys"
Oct 30 19:49:11.770632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.768861    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e6279a-f362-4067-bcdb-59b369f04c71" containerName="revoke-keys"
Oct 30 19:49:11.779183 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.778141    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:11.796228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.795839    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"]
Oct 30 19:49:11.810418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.810359    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-openshift-ca\") pod \"77e6279a-f362-4067-bcdb-59b369f04c71\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") "
Oct 30 19:49:11.811139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.811112    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-xks-ca\") pod \"77e6279a-f362-4067-bcdb-59b369f04c71\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") "
Oct 30 19:49:11.811348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.811317    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e6279a-f362-4067-bcdb-59b369f04c71-kube-api-access-qck4b\" (UniqueName: \"kubernetes.io/projected/77e6279a-f362-4067-bcdb-59b369f04c71-kube-api-access-qck4b\") pod \"77e6279a-f362-4067-bcdb-59b369f04c71\" (UID: \"77e6279a-f362-4067-bcdb-59b369f04c71\") "
Oct 30 19:49:11.819175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.819051    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e6279a-f362-4067-bcdb-59b369f04c71-kube-api-access-qck4b" pod "77e6279a-f362-4067-bcdb-59b369f04c71" (UID: "77e6279a-f362-4067-bcdb-59b369f04c71"). InnerVolumeSpecName "kube-api-access-qck4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:11.824527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.823584    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-openshift-ca" pod "77e6279a-f362-4067-bcdb-59b369f04c71" (UID: "77e6279a-f362-4067-bcdb-59b369f04c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:11.826014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.825980    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-xks-ca" pod "77e6279a-f362-4067-bcdb-59b369f04c71" (UID: "77e6279a-f362-4067-bcdb-59b369f04c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:11.868529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.857020    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn" event={"ID":"77e6279a-f362-4067-bcdb-59b369f04c71","Type":"ContainerDied","Data":"8d8194ea14daa4e7758767a66df1a2b809b43638c2cdc3f056e635453005abcd"}
Oct 30 19:49:11.868529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.857093    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d8194ea14daa4e7758767a66df1a2b809b43638c2cdc3f056e635453005abcd"
Oct 30 19:49:11.868529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.857167    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"
Oct 30 19:49:11.868529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.863117    4519 generic.go:356] "Generic (PLEG): container finished" podID="9f171125-a5d1-4a3a-b6d5-8b56531ef647" containerID="60633fb7f7fb846b2ff9ca8f26d0614c32691ad0701e0cce21590f9162262f40" exitCode=0
Oct 30 19:49:11.868529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.863199    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm" event={"ID":"9f171125-a5d1-4a3a-b6d5-8b56531ef647","Type":"ContainerDied","Data":"60633fb7f7fb846b2ff9ca8f26d0614c32691ad0701e0cce21590f9162262f40"}
Oct 30 19:49:11.913741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.913686    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:11.913944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.913758    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mccb\" (UniqueName: \"kubernetes.io/projected/799bdb9d-dd16-4e35-b92f-fe2699054515-kube-api-access-9mccb\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:11.913997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.913947    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:11.914084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.914037    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:11.914084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.914058    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qck4b\" (UniqueName: \"kubernetes.io/projected/77e6279a-f362-4067-bcdb-59b369f04c71-kube-api-access-qck4b\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:11.914189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:11.914158    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e6279a-f362-4067-bcdb-59b369f04c71-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:12.014949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.014903    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:12.015167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.015007    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:12.015167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.015042    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mccb\" (UniqueName: \"kubernetes.io/projected/799bdb9d-dd16-4e35-b92f-fe2699054515-kube-api-access-9mccb\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:12.015851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.015820    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:12.017670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.017648    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:12.035156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.035075    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mccb\" (UniqueName: \"kubernetes.io/projected/799bdb9d-dd16-4e35-b92f-fe2699054515-kube-api-access-9mccb\") pod \"maas-api-revoke-sub-11da3fe3-q8k74\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:12.043541 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.043509    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"]
Oct 30 19:49:12.047629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.047607    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.067410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.067360    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"]
Oct 30 19:49:12.132764 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.132717    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:12.218065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.217721    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xd9k\" (UniqueName: \"kubernetes.io/projected/e95c3e3a-6249-4907-9252-e04dc15ce93a-kube-api-access-2xd9k\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.218065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.217864    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.218065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.217967    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.318986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.318884    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xd9k\" (UniqueName: \"kubernetes.io/projected/e95c3e3a-6249-4907-9252-e04dc15ce93a-kube-api-access-2xd9k\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.318986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.318938    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.318986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.318981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.319790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.319768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.321567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.321543    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.338193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.338158    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xd9k\" (UniqueName: \"kubernetes.io/projected/e95c3e3a-6249-4907-9252-e04dc15ce93a-kube-api-access-2xd9k\") pod \"maas-api-revoke-sub-c9ce72fc-k8g6q\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.363882 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.363834    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:12.454497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.454444    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"]
Oct 30 19:49:12.457407 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:12.457364    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799bdb9d_dd16_4e35_b92f_fe2699054515.slice/crio-5921a16c6f815dff8aaef4e6bc45d98ddfa6126127017cdd0867e8fb5697be2f WatchSource:0}: Error finding container 5921a16c6f815dff8aaef4e6bc45d98ddfa6126127017cdd0867e8fb5697be2f: Status 404 returned error can't find the container with id 5921a16c6f815dff8aaef4e6bc45d98ddfa6126127017cdd0867e8fb5697be2f
Oct 30 19:49:12.674602 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:12.674565    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95c3e3a_6249_4907_9252_e04dc15ce93a.slice/crio-86cf613848b547f69f89443b14198d80f61da6a45dc5c68b9f4270d90ac29964 WatchSource:0}: Error finding container 86cf613848b547f69f89443b14198d80f61da6a45dc5c68b9f4270d90ac29964: Status 404 returned error can't find the container with id 86cf613848b547f69f89443b14198d80f61da6a45dc5c68b9f4270d90ac29964
Oct 30 19:49:12.677085 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.677050    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"]
Oct 30 19:49:12.876280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.876082    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74" event={"ID":"799bdb9d-dd16-4e35-b92f-fe2699054515","Type":"ContainerStarted","Data":"14bcd90bb90371413dbae5de8b3e377c0d26a0387369639c1c487e362b2fc8df"}
Oct 30 19:49:12.876280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.876187    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74" event={"ID":"799bdb9d-dd16-4e35-b92f-fe2699054515","Type":"ContainerStarted","Data":"5921a16c6f815dff8aaef4e6bc45d98ddfa6126127017cdd0867e8fb5697be2f"}
Oct 30 19:49:12.878397 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.878350    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q" event={"ID":"e95c3e3a-6249-4907-9252-e04dc15ce93a","Type":"ContainerStarted","Data":"86cf613848b547f69f89443b14198d80f61da6a45dc5c68b9f4270d90ac29964"}
Oct 30 19:49:12.920877 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:12.917430    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74" podStartSLOduration=1.917412849 podStartE2EDuration="1.917412849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:12.916251463 +0000 UTC m=+877.416955186" watchObservedRunningTime="2026-10-30 19:49:12.917412849 +0000 UTC m=+877.418116573"
Oct 30 19:49:13.021935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.021904    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:13.126841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.126800    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-openshift-ca\") pod \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") "
Oct 30 19:49:13.127023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.126908    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-xks-ca\") pod \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") "
Oct 30 19:49:13.127090 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.127057    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f171125-a5d1-4a3a-b6d5-8b56531ef647-kube-api-access-8dt8w\" (UniqueName: \"kubernetes.io/projected/9f171125-a5d1-4a3a-b6d5-8b56531ef647-kube-api-access-8dt8w\") pod \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\" (UID: \"9f171125-a5d1-4a3a-b6d5-8b56531ef647\") "
Oct 30 19:49:13.127215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.127189    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-openshift-ca" pod "9f171125-a5d1-4a3a-b6d5-8b56531ef647" (UID: "9f171125-a5d1-4a3a-b6d5-8b56531ef647"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:13.127452 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.127432    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:13.129004 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.128973    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-xks-ca" pod "9f171125-a5d1-4a3a-b6d5-8b56531ef647" (UID: "9f171125-a5d1-4a3a-b6d5-8b56531ef647"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:13.129092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.129067    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f171125-a5d1-4a3a-b6d5-8b56531ef647-kube-api-access-8dt8w" pod "9f171125-a5d1-4a3a-b6d5-8b56531ef647" (UID: "9f171125-a5d1-4a3a-b6d5-8b56531ef647"). InnerVolumeSpecName "kube-api-access-8dt8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:13.233420 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.230319    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dt8w\" (UniqueName: \"kubernetes.io/projected/9f171125-a5d1-4a3a-b6d5-8b56531ef647-kube-api-access-8dt8w\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:13.233420 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.230368    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f171125-a5d1-4a3a-b6d5-8b56531ef647-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:13.888575 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.888448    4519 generic.go:356] "Generic (PLEG): container finished" podID="e95c3e3a-6249-4907-9252-e04dc15ce93a" containerID="add205026179aa0adafd8d3f8ed9719dd810b847cab93ff2975674908422c6af" exitCode=0
Oct 30 19:49:13.888575 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.888506    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q" event={"ID":"e95c3e3a-6249-4907-9252-e04dc15ce93a","Type":"ContainerDied","Data":"add205026179aa0adafd8d3f8ed9719dd810b847cab93ff2975674908422c6af"}
Oct 30 19:49:13.890749 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.890714    4519 generic.go:356] "Generic (PLEG): container finished" podID="799bdb9d-dd16-4e35-b92f-fe2699054515" containerID="14bcd90bb90371413dbae5de8b3e377c0d26a0387369639c1c487e362b2fc8df" exitCode=0
Oct 30 19:49:13.890893 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.890799    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74" event={"ID":"799bdb9d-dd16-4e35-b92f-fe2699054515","Type":"ContainerDied","Data":"14bcd90bb90371413dbae5de8b3e377c0d26a0387369639c1c487e362b2fc8df"}
Oct 30 19:49:13.892282 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.892257    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm" event={"ID":"9f171125-a5d1-4a3a-b6d5-8b56531ef647","Type":"ContainerDied","Data":"1032799ef248b9b63d824d11fa8a4f3b94904f225f163677cabbdf1e81e380ed"}
Oct 30 19:49:13.892382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.892294    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"
Oct 30 19:49:13.892445 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:13.892298    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1032799ef248b9b63d824d11fa8a4f3b94904f225f163677cabbdf1e81e380ed"
Oct 30 19:49:15.080703 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.080395    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:15.113885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.113852    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:15.153272 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.153156    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-xks-ca\") pod \"799bdb9d-dd16-4e35-b92f-fe2699054515\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") "
Oct 30 19:49:15.153442 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.153280    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-openshift-ca\") pod \"799bdb9d-dd16-4e35-b92f-fe2699054515\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") "
Oct 30 19:49:15.154010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.153978    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-openshift-ca" pod "799bdb9d-dd16-4e35-b92f-fe2699054515" (UID: "799bdb9d-dd16-4e35-b92f-fe2699054515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:15.154120 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.154038    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799bdb9d-dd16-4e35-b92f-fe2699054515-kube-api-access-9mccb\" (UniqueName: \"kubernetes.io/projected/799bdb9d-dd16-4e35-b92f-fe2699054515-kube-api-access-9mccb\") pod \"799bdb9d-dd16-4e35-b92f-fe2699054515\" (UID: \"799bdb9d-dd16-4e35-b92f-fe2699054515\") "
Oct 30 19:49:15.154193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.154121    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95c3e3a-6249-4907-9252-e04dc15ce93a-kube-api-access-2xd9k\" (UniqueName: \"kubernetes.io/projected/e95c3e3a-6249-4907-9252-e04dc15ce93a-kube-api-access-2xd9k\") pod \"e95c3e3a-6249-4907-9252-e04dc15ce93a\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") "
Oct 30 19:49:15.154333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.154286    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-xks-ca\") pod \"e95c3e3a-6249-4907-9252-e04dc15ce93a\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") "
Oct 30 19:49:15.154720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.154642    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-openshift-ca\") pod \"e95c3e3a-6249-4907-9252-e04dc15ce93a\" (UID: \"e95c3e3a-6249-4907-9252-e04dc15ce93a\") "
Oct 30 19:49:15.155722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.155212    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:15.155722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.155689    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-openshift-ca" pod "e95c3e3a-6249-4907-9252-e04dc15ce93a" (UID: "e95c3e3a-6249-4907-9252-e04dc15ce93a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:15.155722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.156565    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-xks-ca" pod "799bdb9d-dd16-4e35-b92f-fe2699054515" (UID: "799bdb9d-dd16-4e35-b92f-fe2699054515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:15.155722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.157048    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95c3e3a-6249-4907-9252-e04dc15ce93a-kube-api-access-2xd9k" pod "e95c3e3a-6249-4907-9252-e04dc15ce93a" (UID: "e95c3e3a-6249-4907-9252-e04dc15ce93a"). InnerVolumeSpecName "kube-api-access-2xd9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.155722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.157258    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799bdb9d-dd16-4e35-b92f-fe2699054515-kube-api-access-9mccb" pod "799bdb9d-dd16-4e35-b92f-fe2699054515" (UID: "799bdb9d-dd16-4e35-b92f-fe2699054515"). InnerVolumeSpecName "kube-api-access-9mccb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.155722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.158180    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-xks-ca" pod "e95c3e3a-6249-4907-9252-e04dc15ce93a" (UID: "e95c3e3a-6249-4907-9252-e04dc15ce93a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:15.256341 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.256297    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:15.256341 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.256335    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95c3e3a-6249-4907-9252-e04dc15ce93a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:15.256341 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.256345    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799bdb9d-dd16-4e35-b92f-fe2699054515-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:15.256341 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.256354    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mccb\" (UniqueName: \"kubernetes.io/projected/799bdb9d-dd16-4e35-b92f-fe2699054515-kube-api-access-9mccb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:15.256661 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.256363    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xd9k\" (UniqueName: \"kubernetes.io/projected/e95c3e3a-6249-4907-9252-e04dc15ce93a-kube-api-access-2xd9k\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:15.907671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.907569    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"
Oct 30 19:49:15.907671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.907602    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q" event={"ID":"e95c3e3a-6249-4907-9252-e04dc15ce93a","Type":"ContainerDied","Data":"86cf613848b547f69f89443b14198d80f61da6a45dc5c68b9f4270d90ac29964"}
Oct 30 19:49:15.907898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.907690    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86cf613848b547f69f89443b14198d80f61da6a45dc5c68b9f4270d90ac29964"
Oct 30 19:49:15.909193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.909167    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"
Oct 30 19:49:15.909327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.909170    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74" event={"ID":"799bdb9d-dd16-4e35-b92f-fe2699054515","Type":"ContainerDied","Data":"5921a16c6f815dff8aaef4e6bc45d98ddfa6126127017cdd0867e8fb5697be2f"}
Oct 30 19:49:15.909327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:15.909297    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5921a16c6f815dff8aaef4e6bc45d98ddfa6126127017cdd0867e8fb5697be2f"
Oct 30 19:49:19.285403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.285365    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"]
Oct 30 19:49:19.285872 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.285857    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e95c3e3a-6249-4907-9252-e04dc15ce93a" containerName="revoke-keys"
Oct 30 19:49:19.285910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.285874    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95c3e3a-6249-4907-9252-e04dc15ce93a" containerName="revoke-keys"
Oct 30 19:49:19.285910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.285892    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9f171125-a5d1-4a3a-b6d5-8b56531ef647" containerName="revoke-keys"
Oct 30 19:49:19.285910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.285897    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f171125-a5d1-4a3a-b6d5-8b56531ef647" containerName="revoke-keys"
Oct 30 19:49:19.285910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.285908    4519 cpu_manager.go:414] "containerMap: removing container" podUID="799bdb9d-dd16-4e35-b92f-fe2699054515" containerName="revoke-keys"
Oct 30 19:49:19.286048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.285913    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799bdb9d-dd16-4e35-b92f-fe2699054515" containerName="revoke-keys"
Oct 30 19:49:19.286048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.286024    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95c3e3a-6249-4907-9252-e04dc15ce93a" containerName="revoke-keys"
Oct 30 19:49:19.286048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.286033    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f171125-a5d1-4a3a-b6d5-8b56531ef647" containerName="revoke-keys"
Oct 30 19:49:19.286048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.286039    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="799bdb9d-dd16-4e35-b92f-fe2699054515" containerName="revoke-keys"
Oct 30 19:49:19.290736 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.290714    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.320656 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.320618    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:49:19.320861 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.320728    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:19.343556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.343514    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"]
Oct 30 19:49:19.409877 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.409836    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.410046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.409884    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.410046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.409910    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzb52\" (UniqueName: \"kubernetes.io/projected/3bc7717d-3b98-4b2b-bf93-2d6194a73452-kube-api-access-gzb52\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.510831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.510784    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.511022 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.510849    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.511022 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.510876    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzb52\" (UniqueName: \"kubernetes.io/projected/3bc7717d-3b98-4b2b-bf93-2d6194a73452-kube-api-access-gzb52\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.511549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.511518    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.513531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.513502    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.531017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.530978    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzb52\" (UniqueName: \"kubernetes.io/projected/3bc7717d-3b98-4b2b-bf93-2d6194a73452-kube-api-access-gzb52\") pod \"maas-api-revoke-sub-7a52caf0-f7vmg\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.608023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.607928    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:19.916505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.916451    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"]
Oct 30 19:49:19.922080 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.922046    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:19.945534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.944848    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"]
Oct 30 19:49:19.958897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:19.958865    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"]
Oct 30 19:49:20.022947 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.022908    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.023146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.023115    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.023191 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.023164    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px89n\" (UniqueName: \"kubernetes.io/projected/09003d52-027e-4859-8e6e-52b9e0c15daa-kube-api-access-px89n\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.123978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.123941    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.124151 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.124110    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.124251 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.124232    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px89n\" (UniqueName: \"kubernetes.io/projected/09003d52-027e-4859-8e6e-52b9e0c15daa-kube-api-access-px89n\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.124859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.124831    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.147602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.147568    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.148016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.147992    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px89n\" (UniqueName: \"kubernetes.io/projected/09003d52-027e-4859-8e6e-52b9e0c15daa-kube-api-access-px89n\") pod \"maas-api-revoke-sub-25f068df-2k2nk\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.246769 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.246662    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:20.564240 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:20.564203    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09003d52_027e_4859_8e6e_52b9e0c15daa.slice/crio-7821bd63de7544c454c6adbab51a05a51ffe6a938d981f18a0bef2ff086ff4d5 WatchSource:0}: Error finding container 7821bd63de7544c454c6adbab51a05a51ffe6a938d981f18a0bef2ff086ff4d5: Status 404 returned error can't find the container with id 7821bd63de7544c454c6adbab51a05a51ffe6a938d981f18a0bef2ff086ff4d5
Oct 30 19:49:20.566739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.566709    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"]
Oct 30 19:49:20.957371 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.957336    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk" event={"ID":"09003d52-027e-4859-8e6e-52b9e0c15daa","Type":"ContainerStarted","Data":"1b5375b5071d4f72e7ec20fc4a0e74557ebebab3718958030b855872ecdfd174"}
Oct 30 19:49:20.957610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.957390    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk" event={"ID":"09003d52-027e-4859-8e6e-52b9e0c15daa","Type":"ContainerStarted","Data":"7821bd63de7544c454c6adbab51a05a51ffe6a938d981f18a0bef2ff086ff4d5"}
Oct 30 19:49:20.959083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.959053    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg" event={"ID":"3bc7717d-3b98-4b2b-bf93-2d6194a73452","Type":"ContainerStarted","Data":"79c307fc4aadb09acbf98d1bbf7940c0316ab5f7da1063f1fbef7e7c464e56d4"}
Oct 30 19:49:20.959208 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.959106    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg" event={"ID":"3bc7717d-3b98-4b2b-bf93-2d6194a73452","Type":"ContainerStarted","Data":"ed025a10bb56661c4d70928ac5448c580b6b1b92902513ccc1829cce164710b4"}
Oct 30 19:49:20.999046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:20.998991    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk" podStartSLOduration=1.9989738479999999 podStartE2EDuration="1.998973848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:20.997151051 +0000 UTC m=+885.497854775" watchObservedRunningTime="2026-10-30 19:49:20.998973848 +0000 UTC m=+885.499677631"
Oct 30 19:49:21.038895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:21.038837    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg" podStartSLOduration=2.038819984 podStartE2EDuration="2.038819984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:21.036630973 +0000 UTC m=+885.537334709" watchObservedRunningTime="2026-10-30 19:49:21.038819984 +0000 UTC m=+885.539523707"
Oct 30 19:49:21.968478 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:21.968435    4519 generic.go:356] "Generic (PLEG): container finished" podID="09003d52-027e-4859-8e6e-52b9e0c15daa" containerID="1b5375b5071d4f72e7ec20fc4a0e74557ebebab3718958030b855872ecdfd174" exitCode=0
Oct 30 19:49:21.968927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:21.968529    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk" event={"ID":"09003d52-027e-4859-8e6e-52b9e0c15daa","Type":"ContainerDied","Data":"1b5375b5071d4f72e7ec20fc4a0e74557ebebab3718958030b855872ecdfd174"}
Oct 30 19:49:21.970743 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:21.970706    4519 generic.go:356] "Generic (PLEG): container finished" podID="3bc7717d-3b98-4b2b-bf93-2d6194a73452" containerID="79c307fc4aadb09acbf98d1bbf7940c0316ab5f7da1063f1fbef7e7c464e56d4" exitCode=0
Oct 30 19:49:21.970903 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:21.970797    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg" event={"ID":"3bc7717d-3b98-4b2b-bf93-2d6194a73452","Type":"ContainerDied","Data":"79c307fc4aadb09acbf98d1bbf7940c0316ab5f7da1063f1fbef7e7c464e56d4"}
Oct 30 19:49:22.212912 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.212870    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"]
Oct 30 19:49:22.217346 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.217317    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.237110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.237073    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"]
Oct 30 19:49:22.325031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.324995    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"]
Oct 30 19:49:22.329497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.329454    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.351298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.351257    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"]
Oct 30 19:49:22.357123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.357089    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.357331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.357129    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzgpp\" (UniqueName: \"kubernetes.io/projected/f378ee03-dabe-4875-aaa1-104d8af75689-kube-api-access-vzgpp\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.357331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.357286    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.458745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.458703    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.458745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.458747    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.459015 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.458776    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzgpp\" (UniqueName: \"kubernetes.io/projected/f378ee03-dabe-4875-aaa1-104d8af75689-kube-api-access-vzgpp\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.459015 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.458947    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.459132 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.459010    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.459189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.459149    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzfx\" (UniqueName: \"kubernetes.io/projected/95e2ac64-9211-48c5-9b37-e748f83b9f45-kube-api-access-sfzfx\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.459682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.459653    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.461326 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.461301    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.491693 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.486200    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzgpp\" (UniqueName: \"kubernetes.io/projected/f378ee03-dabe-4875-aaa1-104d8af75689-kube-api-access-vzgpp\") pod \"maas-api-revoke-sub-99fb68e6-f27kj\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.531693 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.531640    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:22.560184 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.560138    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.560355 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.560322    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzfx\" (UniqueName: \"kubernetes.io/projected/95e2ac64-9211-48c5-9b37-e748f83b9f45-kube-api-access-sfzfx\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.560413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.560369    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.561296 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.561264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.563569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.563533    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.581806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.581766    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzfx\" (UniqueName: \"kubernetes.io/projected/95e2ac64-9211-48c5-9b37-e748f83b9f45-kube-api-access-sfzfx\") pod \"maas-api-revoke-sub-2c2a96a4-b9nhr\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.644657 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.644623    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:22.862693 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:22.862653    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf378ee03_dabe_4875_aaa1_104d8af75689.slice/crio-6433e9395888ad70ea88defc38eb9a0f434e7eb4ea7f2b28b017d41a5cc82761 WatchSource:0}: Error finding container 6433e9395888ad70ea88defc38eb9a0f434e7eb4ea7f2b28b017d41a5cc82761: Status 404 returned error can't find the container with id 6433e9395888ad70ea88defc38eb9a0f434e7eb4ea7f2b28b017d41a5cc82761
Oct 30 19:49:22.862862 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.862844    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"]
Oct 30 19:49:22.956612 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:22.956573    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e2ac64_9211_48c5_9b37_e748f83b9f45.slice/crio-3ee8f3aca1f2491c7ea45568020eeb25acfaa6b584e8b68ef7c662b4c565ed58 WatchSource:0}: Error finding container 3ee8f3aca1f2491c7ea45568020eeb25acfaa6b584e8b68ef7c662b4c565ed58: Status 404 returned error can't find the container with id 3ee8f3aca1f2491c7ea45568020eeb25acfaa6b584e8b68ef7c662b4c565ed58
Oct 30 19:49:22.958410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.958306    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"]
Oct 30 19:49:22.982415 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.982376    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr" event={"ID":"95e2ac64-9211-48c5-9b37-e748f83b9f45","Type":"ContainerStarted","Data":"3ee8f3aca1f2491c7ea45568020eeb25acfaa6b584e8b68ef7c662b4c565ed58"}
Oct 30 19:49:22.983775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:22.983740    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj" event={"ID":"f378ee03-dabe-4875-aaa1-104d8af75689","Type":"ContainerStarted","Data":"6433e9395888ad70ea88defc38eb9a0f434e7eb4ea7f2b28b017d41a5cc82761"}
Oct 30 19:49:23.140501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.140443    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:23.175601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.175568    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:23.268495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.268443    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-xks-ca\") pod \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") "
Oct 30 19:49:23.268682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.268507    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc7717d-3b98-4b2b-bf93-2d6194a73452-kube-api-access-gzb52\" (UniqueName: \"kubernetes.io/projected/3bc7717d-3b98-4b2b-bf93-2d6194a73452-kube-api-access-gzb52\") pod \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") "
Oct 30 19:49:23.268682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.268562    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09003d52-027e-4859-8e6e-52b9e0c15daa-kube-api-access-px89n\" (UniqueName: \"kubernetes.io/projected/09003d52-027e-4859-8e6e-52b9e0c15daa-kube-api-access-px89n\") pod \"09003d52-027e-4859-8e6e-52b9e0c15daa\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") "
Oct 30 19:49:23.268682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.268647    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-openshift-ca\") pod \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\" (UID: \"3bc7717d-3b98-4b2b-bf93-2d6194a73452\") "
Oct 30 19:49:23.268863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.268746    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-xks-ca\") pod \"09003d52-027e-4859-8e6e-52b9e0c15daa\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") "
Oct 30 19:49:23.268863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.268774    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-openshift-ca\") pod \"09003d52-027e-4859-8e6e-52b9e0c15daa\" (UID: \"09003d52-027e-4859-8e6e-52b9e0c15daa\") "
Oct 30 19:49:23.269288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.269254    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-openshift-ca" pod "3bc7717d-3b98-4b2b-bf93-2d6194a73452" (UID: "3bc7717d-3b98-4b2b-bf93-2d6194a73452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:23.269764 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.269727    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-openshift-ca" pod "09003d52-027e-4859-8e6e-52b9e0c15daa" (UID: "09003d52-027e-4859-8e6e-52b9e0c15daa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:23.271746 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.271677    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-xks-ca" pod "3bc7717d-3b98-4b2b-bf93-2d6194a73452" (UID: "3bc7717d-3b98-4b2b-bf93-2d6194a73452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:23.272589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.272545    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-xks-ca" pod "09003d52-027e-4859-8e6e-52b9e0c15daa" (UID: "09003d52-027e-4859-8e6e-52b9e0c15daa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:23.272797 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.272772    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc7717d-3b98-4b2b-bf93-2d6194a73452-kube-api-access-gzb52" pod "3bc7717d-3b98-4b2b-bf93-2d6194a73452" (UID: "3bc7717d-3b98-4b2b-bf93-2d6194a73452"). InnerVolumeSpecName "kube-api-access-gzb52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:23.272797 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.272785    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09003d52-027e-4859-8e6e-52b9e0c15daa-kube-api-access-px89n" pod "09003d52-027e-4859-8e6e-52b9e0c15daa" (UID: "09003d52-027e-4859-8e6e-52b9e0c15daa"). InnerVolumeSpecName "kube-api-access-px89n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:23.370617 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.370577    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:23.370617 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.370617    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09003d52-027e-4859-8e6e-52b9e0c15daa-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:23.370822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.370636    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:23.370822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.370653    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzb52\" (UniqueName: \"kubernetes.io/projected/3bc7717d-3b98-4b2b-bf93-2d6194a73452-kube-api-access-gzb52\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:23.370822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.370668    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px89n\" (UniqueName: \"kubernetes.io/projected/09003d52-027e-4859-8e6e-52b9e0c15daa-kube-api-access-px89n\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:23.370822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.370682    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc7717d-3b98-4b2b-bf93-2d6194a73452-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:23.992997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.992960    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"
Oct 30 19:49:23.993399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.993034    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk" event={"ID":"09003d52-027e-4859-8e6e-52b9e0c15daa","Type":"ContainerDied","Data":"7821bd63de7544c454c6adbab51a05a51ffe6a938d981f18a0bef2ff086ff4d5"}
Oct 30 19:49:23.993399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.993089    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7821bd63de7544c454c6adbab51a05a51ffe6a938d981f18a0bef2ff086ff4d5"
Oct 30 19:49:23.994829 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.994807    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"
Oct 30 19:49:23.994972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.994829    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg" event={"ID":"3bc7717d-3b98-4b2b-bf93-2d6194a73452","Type":"ContainerDied","Data":"ed025a10bb56661c4d70928ac5448c580b6b1b92902513ccc1829cce164710b4"}
Oct 30 19:49:23.994972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.994879    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed025a10bb56661c4d70928ac5448c580b6b1b92902513ccc1829cce164710b4"
Oct 30 19:49:23.997338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.997310    4519 generic.go:356] "Generic (PLEG): container finished" podID="f378ee03-dabe-4875-aaa1-104d8af75689" containerID="08fd9f01ae55d49fe6ab2417605148545081787dab1b95e89383114b2e1a6958" exitCode=0
Oct 30 19:49:23.997453 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.997388    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj" event={"ID":"f378ee03-dabe-4875-aaa1-104d8af75689","Type":"ContainerDied","Data":"08fd9f01ae55d49fe6ab2417605148545081787dab1b95e89383114b2e1a6958"}
Oct 30 19:49:23.999225 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.999146    4519 generic.go:356] "Generic (PLEG): container finished" podID="95e2ac64-9211-48c5-9b37-e748f83b9f45" containerID="5fc9fb86bca25d6131ad630f3ab56d1e7abbfa9a944dcc963f177901a4881d12" exitCode=0
Oct 30 19:49:23.999357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:23.999220    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr" event={"ID":"95e2ac64-9211-48c5-9b37-e748f83b9f45","Type":"ContainerDied","Data":"5fc9fb86bca25d6131ad630f3ab56d1e7abbfa9a944dcc963f177901a4881d12"}
Oct 30 19:49:25.167507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.167440    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"]
Oct 30 19:49:25.168167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.168143    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3bc7717d-3b98-4b2b-bf93-2d6194a73452" containerName="revoke-keys"
Oct 30 19:49:25.168167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.168166    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc7717d-3b98-4b2b-bf93-2d6194a73452" containerName="revoke-keys"
Oct 30 19:49:25.168313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.168174    4519 cpu_manager.go:414] "containerMap: removing container" podUID="09003d52-027e-4859-8e6e-52b9e0c15daa" containerName="revoke-keys"
Oct 30 19:49:25.168313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.168179    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09003d52-027e-4859-8e6e-52b9e0c15daa" containerName="revoke-keys"
Oct 30 19:49:25.168313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.168274    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc7717d-3b98-4b2b-bf93-2d6194a73452" containerName="revoke-keys"
Oct 30 19:49:25.168313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.168285    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="09003d52-027e-4859-8e6e-52b9e0c15daa" containerName="revoke-keys"
Oct 30 19:49:25.171970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.171942    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.187842 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.187729    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:25.191306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.191266    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"]
Oct 30 19:49:25.229986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.229958    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:25.304100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.301225    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e2ac64-9211-48c5-9b37-e748f83b9f45-kube-api-access-sfzfx\" (UniqueName: \"kubernetes.io/projected/95e2ac64-9211-48c5-9b37-e748f83b9f45-kube-api-access-sfzfx\") pod \"95e2ac64-9211-48c5-9b37-e748f83b9f45\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") "
Oct 30 19:49:25.312912 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.312866    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-openshift-ca\") pod \"95e2ac64-9211-48c5-9b37-e748f83b9f45\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") "
Oct 30 19:49:25.313120 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.313043    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-openshift-ca\") pod \"f378ee03-dabe-4875-aaa1-104d8af75689\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") "
Oct 30 19:49:25.313120 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.313112    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-xks-ca\") pod \"f378ee03-dabe-4875-aaa1-104d8af75689\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") "
Oct 30 19:49:25.313236 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.313172    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-xks-ca\") pod \"95e2ac64-9211-48c5-9b37-e748f83b9f45\" (UID: \"95e2ac64-9211-48c5-9b37-e748f83b9f45\") "
Oct 30 19:49:25.313296 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.313239    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f378ee03-dabe-4875-aaa1-104d8af75689-kube-api-access-vzgpp\" (UniqueName: \"kubernetes.io/projected/f378ee03-dabe-4875-aaa1-104d8af75689-kube-api-access-vzgpp\") pod \"f378ee03-dabe-4875-aaa1-104d8af75689\" (UID: \"f378ee03-dabe-4875-aaa1-104d8af75689\") "
Oct 30 19:49:25.313620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.313595    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/10ce29d4-49e9-438d-beca-1364d75a7bd9-kube-api-access-465d4\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.313902 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.313883    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.313967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.313944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.314574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.314525    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-openshift-ca" pod "95e2ac64-9211-48c5-9b37-e748f83b9f45" (UID: "95e2ac64-9211-48c5-9b37-e748f83b9f45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:25.315317 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.315285    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-openshift-ca" pod "f378ee03-dabe-4875-aaa1-104d8af75689" (UID: "f378ee03-dabe-4875-aaa1-104d8af75689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:25.318312 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.318275    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-xks-ca" pod "f378ee03-dabe-4875-aaa1-104d8af75689" (UID: "f378ee03-dabe-4875-aaa1-104d8af75689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:25.318455 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.318439    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f378ee03-dabe-4875-aaa1-104d8af75689-kube-api-access-vzgpp" pod "f378ee03-dabe-4875-aaa1-104d8af75689" (UID: "f378ee03-dabe-4875-aaa1-104d8af75689"). InnerVolumeSpecName "kube-api-access-vzgpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:25.318990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.318963    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-xks-ca" pod "95e2ac64-9211-48c5-9b37-e748f83b9f45" (UID: "95e2ac64-9211-48c5-9b37-e748f83b9f45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:25.319315 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.319285    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e2ac64-9211-48c5-9b37-e748f83b9f45-kube-api-access-sfzfx" pod "95e2ac64-9211-48c5-9b37-e748f83b9f45" (UID: "95e2ac64-9211-48c5-9b37-e748f83b9f45"). InnerVolumeSpecName "kube-api-access-sfzfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:25.415411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415369    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.415411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415421    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.415678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415459    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/10ce29d4-49e9-438d-beca-1364d75a7bd9-kube-api-access-465d4\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.415678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415599    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:25.415678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415612    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f378ee03-dabe-4875-aaa1-104d8af75689-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:25.415678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415625    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:25.415678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415637    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzgpp\" (UniqueName: \"kubernetes.io/projected/f378ee03-dabe-4875-aaa1-104d8af75689-kube-api-access-vzgpp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:25.415678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415651    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzfx\" (UniqueName: \"kubernetes.io/projected/95e2ac64-9211-48c5-9b37-e748f83b9f45-kube-api-access-sfzfx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:25.415678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.415670    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e2ac64-9211-48c5-9b37-e748f83b9f45-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:25.416264 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.416242    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.417983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.417957    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.436586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.436551    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/10ce29d4-49e9-438d-beca-1364d75a7bd9-kube-api-access-465d4\") pod \"maas-api-revoke-sub-c8f55efe-bxktb\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.507672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.507626    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:25.815898 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:25.815816    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ce29d4_49e9_438d_beca_1364d75a7bd9.slice/crio-306a505853f731b4799493ece0c2762eb964d248a0cab752d1f769cd818f0775 WatchSource:0}: Error finding container 306a505853f731b4799493ece0c2762eb964d248a0cab752d1f769cd818f0775: Status 404 returned error can't find the container with id 306a505853f731b4799493ece0c2762eb964d248a0cab752d1f769cd818f0775
Oct 30 19:49:25.818181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:25.817201    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"]
Oct 30 19:49:26.015172 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:26.015131    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb" event={"ID":"10ce29d4-49e9-438d-beca-1364d75a7bd9","Type":"ContainerStarted","Data":"306a505853f731b4799493ece0c2762eb964d248a0cab752d1f769cd818f0775"}
Oct 30 19:49:26.016827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:26.016801    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"
Oct 30 19:49:26.016960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:26.016844    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj" event={"ID":"f378ee03-dabe-4875-aaa1-104d8af75689","Type":"ContainerDied","Data":"6433e9395888ad70ea88defc38eb9a0f434e7eb4ea7f2b28b017d41a5cc82761"}
Oct 30 19:49:26.016960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:26.016902    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6433e9395888ad70ea88defc38eb9a0f434e7eb4ea7f2b28b017d41a5cc82761"
Oct 30 19:49:26.018613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:26.018590    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"
Oct 30 19:49:26.018711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:26.018611    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr" event={"ID":"95e2ac64-9211-48c5-9b37-e748f83b9f45","Type":"ContainerDied","Data":"3ee8f3aca1f2491c7ea45568020eeb25acfaa6b584e8b68ef7c662b4c565ed58"}
Oct 30 19:49:26.018711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:26.018660    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee8f3aca1f2491c7ea45568020eeb25acfaa6b584e8b68ef7c662b4c565ed58"
Oct 30 19:49:27.190423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:27.189663    4519 generic.go:356] "Generic (PLEG): container finished" podID="10ce29d4-49e9-438d-beca-1364d75a7bd9" containerID="9d74fe23b15b630116ae444769a01a02a990a8d9a2836c0bee06ec36a0b4746d" exitCode=0
Oct 30 19:49:27.190423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:27.189967    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb" event={"ID":"10ce29d4-49e9-438d-beca-1364d75a7bd9","Type":"ContainerDied","Data":"9d74fe23b15b630116ae444769a01a02a990a8d9a2836c0bee06ec36a0b4746d"}
Oct 30 19:49:28.347929 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.347898    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:28.419426 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.419387    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-openshift-ca\") pod \"10ce29d4-49e9-438d-beca-1364d75a7bd9\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") "
Oct 30 19:49:28.419614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.419431    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-xks-ca\") pod \"10ce29d4-49e9-438d-beca-1364d75a7bd9\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") "
Oct 30 19:49:28.419614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.419600    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ce29d4-49e9-438d-beca-1364d75a7bd9-kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/10ce29d4-49e9-438d-beca-1364d75a7bd9-kube-api-access-465d4\") pod \"10ce29d4-49e9-438d-beca-1364d75a7bd9\" (UID: \"10ce29d4-49e9-438d-beca-1364d75a7bd9\") "
Oct 30 19:49:28.419902 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.419875    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-openshift-ca" pod "10ce29d4-49e9-438d-beca-1364d75a7bd9" (UID: "10ce29d4-49e9-438d-beca-1364d75a7bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:28.420186 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.420158    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:28.422092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.422052    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-xks-ca" pod "10ce29d4-49e9-438d-beca-1364d75a7bd9" (UID: "10ce29d4-49e9-438d-beca-1364d75a7bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:28.422216 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.422056    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ce29d4-49e9-438d-beca-1364d75a7bd9-kube-api-access-465d4" pod "10ce29d4-49e9-438d-beca-1364d75a7bd9" (UID: "10ce29d4-49e9-438d-beca-1364d75a7bd9"). InnerVolumeSpecName "kube-api-access-465d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:28.521165 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.521073    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/10ce29d4-49e9-438d-beca-1364d75a7bd9-kube-api-access-465d4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:28.521165 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:28.521108    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce29d4-49e9-438d-beca-1364d75a7bd9-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:29.206814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:29.206779    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"
Oct 30 19:49:29.207000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:29.206818    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb" event={"ID":"10ce29d4-49e9-438d-beca-1364d75a7bd9","Type":"ContainerDied","Data":"306a505853f731b4799493ece0c2762eb964d248a0cab752d1f769cd818f0775"}
Oct 30 19:49:29.207000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:29.206904    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="306a505853f731b4799493ece0c2762eb964d248a0cab752d1f769cd818f0775"
Oct 30 19:49:30.298055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298018    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"]
Oct 30 19:49:30.298586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298530    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f378ee03-dabe-4875-aaa1-104d8af75689" containerName="revoke-keys"
Oct 30 19:49:30.298586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298543    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f378ee03-dabe-4875-aaa1-104d8af75689" containerName="revoke-keys"
Oct 30 19:49:30.298586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298562    4519 cpu_manager.go:414] "containerMap: removing container" podUID="10ce29d4-49e9-438d-beca-1364d75a7bd9" containerName="revoke-keys"
Oct 30 19:49:30.298586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298567    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ce29d4-49e9-438d-beca-1364d75a7bd9" containerName="revoke-keys"
Oct 30 19:49:30.298586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298578    4519 cpu_manager.go:414] "containerMap: removing container" podUID="95e2ac64-9211-48c5-9b37-e748f83b9f45" containerName="revoke-keys"
Oct 30 19:49:30.298586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298583    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e2ac64-9211-48c5-9b37-e748f83b9f45" containerName="revoke-keys"
Oct 30 19:49:30.298778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298663    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f378ee03-dabe-4875-aaa1-104d8af75689" containerName="revoke-keys"
Oct 30 19:49:30.298778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298672    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ce29d4-49e9-438d-beca-1364d75a7bd9" containerName="revoke-keys"
Oct 30 19:49:30.298778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.298678    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e2ac64-9211-48c5-9b37-e748f83b9f45" containerName="revoke-keys"
Oct 30 19:49:30.302479 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.302446    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.336234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.336202    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:49:30.343833 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.343799    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:30.345272 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.345244    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.345410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.345359    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.345410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.345402    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5tjg\" (UniqueName: \"kubernetes.io/projected/e78851ec-275d-4750-8e84-4a3f0e16bc85-kube-api-access-j5tjg\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.354656 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.354614    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"]
Oct 30 19:49:30.448368 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.448300    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5tjg\" (UniqueName: \"kubernetes.io/projected/e78851ec-275d-4750-8e84-4a3f0e16bc85-kube-api-access-j5tjg\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.448663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.448570    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.449387 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.449362    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.450387 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.450361    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.452576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.452550    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.477315 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.477271    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5tjg\" (UniqueName: \"kubernetes.io/projected/e78851ec-275d-4750-8e84-4a3f0e16bc85-kube-api-access-j5tjg\") pod \"maas-api-revoke-sub-d4cfbff9-qggvg\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.619184 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.619095    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:30.936648 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:30.936603    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78851ec_275d_4750_8e84_4a3f0e16bc85.slice/crio-817d5b144187f6a8e40253567354ca3b9c6919fc2103bf5abe99003778b45ae8 WatchSource:0}: Error finding container 817d5b144187f6a8e40253567354ca3b9c6919fc2103bf5abe99003778b45ae8: Status 404 returned error can't find the container with id 817d5b144187f6a8e40253567354ca3b9c6919fc2103bf5abe99003778b45ae8
Oct 30 19:49:30.937241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:30.937214    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"]
Oct 30 19:49:31.222710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:31.222612    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg" event={"ID":"e78851ec-275d-4750-8e84-4a3f0e16bc85","Type":"ContainerStarted","Data":"817d5b144187f6a8e40253567354ca3b9c6919fc2103bf5abe99003778b45ae8"}
Oct 30 19:49:32.222528 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:49:32.222491    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:49:32.232294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:32.232254    4519 generic.go:356] "Generic (PLEG): container finished" podID="e78851ec-275d-4750-8e84-4a3f0e16bc85" containerID="eacf0d5c83202b22eb46124b6f53fe95f89ac9e03ea67c91a816e86a86dd25e2" exitCode=0
Oct 30 19:49:32.232533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:32.232415    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg" event={"ID":"e78851ec-275d-4750-8e84-4a3f0e16bc85","Type":"ContainerDied","Data":"eacf0d5c83202b22eb46124b6f53fe95f89ac9e03ea67c91a816e86a86dd25e2"}
Oct 30 19:49:33.394791 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.394760    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:33.486362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.486323    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78851ec-275d-4750-8e84-4a3f0e16bc85-kube-api-access-j5tjg\" (UniqueName: \"kubernetes.io/projected/e78851ec-275d-4750-8e84-4a3f0e16bc85-kube-api-access-j5tjg\") pod \"e78851ec-275d-4750-8e84-4a3f0e16bc85\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") "
Oct 30 19:49:33.486604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.486410    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-xks-ca\") pod \"e78851ec-275d-4750-8e84-4a3f0e16bc85\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") "
Oct 30 19:49:33.486604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.486487    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-openshift-ca\") pod \"e78851ec-275d-4750-8e84-4a3f0e16bc85\" (UID: \"e78851ec-275d-4750-8e84-4a3f0e16bc85\") "
Oct 30 19:49:33.486937 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.486907    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-openshift-ca" pod "e78851ec-275d-4750-8e84-4a3f0e16bc85" (UID: "e78851ec-275d-4750-8e84-4a3f0e16bc85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:33.488712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.488677    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-xks-ca" pod "e78851ec-275d-4750-8e84-4a3f0e16bc85" (UID: "e78851ec-275d-4750-8e84-4a3f0e16bc85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:33.488822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.488732    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78851ec-275d-4750-8e84-4a3f0e16bc85-kube-api-access-j5tjg" pod "e78851ec-275d-4750-8e84-4a3f0e16bc85" (UID: "e78851ec-275d-4750-8e84-4a3f0e16bc85"). InnerVolumeSpecName "kube-api-access-j5tjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:33.588308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.588266    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:33.588308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.588302    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5tjg\" (UniqueName: \"kubernetes.io/projected/e78851ec-275d-4750-8e84-4a3f0e16bc85-kube-api-access-j5tjg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:33.588308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:33.588313    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78851ec-275d-4750-8e84-4a3f0e16bc85-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.250968 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:34.250937    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"
Oct 30 19:49:34.251129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:34.250938    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg" event={"ID":"e78851ec-275d-4750-8e84-4a3f0e16bc85","Type":"ContainerDied","Data":"817d5b144187f6a8e40253567354ca3b9c6919fc2103bf5abe99003778b45ae8"}
Oct 30 19:49:34.251129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:34.251068    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817d5b144187f6a8e40253567354ca3b9c6919fc2103bf5abe99003778b45ae8"
Oct 30 19:49:37.342029 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.341983    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"]
Oct 30 19:49:37.342548 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.342532    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e78851ec-275d-4750-8e84-4a3f0e16bc85" containerName="revoke-keys"
Oct 30 19:49:37.342598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.342549    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78851ec-275d-4750-8e84-4a3f0e16bc85" containerName="revoke-keys"
Oct 30 19:49:37.342642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.342633    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78851ec-275d-4750-8e84-4a3f0e16bc85" containerName="revoke-keys"
Oct 30 19:49:37.346592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.346563    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.369455 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.369412    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"]
Oct 30 19:49:37.375365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.375218    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:37.375365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.375336    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:49:37.434216 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.434169    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.434382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.434241    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4cc\" (UniqueName: \"kubernetes.io/projected/acce9ca9-bca5-4932-9ceb-35cc4bd63055-kube-api-access-8p4cc\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.434576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.434542    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.536303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.536257    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.536534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.536326    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.536534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.536383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4cc\" (UniqueName: \"kubernetes.io/projected/acce9ca9-bca5-4932-9ceb-35cc4bd63055-kube-api-access-8p4cc\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.537093 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.537065    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.538759 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.538739    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.557430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.557392    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4cc\" (UniqueName: \"kubernetes.io/projected/acce9ca9-bca5-4932-9ceb-35cc4bd63055-kube-api-access-8p4cc\") pod \"maas-api-revoke-sub-db2c89ee-9gvnb\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.663189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.663139    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:37.994717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:37.994671    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"]
Oct 30 19:49:37.998031 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:37.997993    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacce9ca9_bca5_4932_9ceb_35cc4bd63055.slice/crio-5148b28ae280afa0b2eb33aa868ff64e3e87502cd63a8ca881a4380c0b4e07e8 WatchSource:0}: Error finding container 5148b28ae280afa0b2eb33aa868ff64e3e87502cd63a8ca881a4380c0b4e07e8: Status 404 returned error can't find the container with id 5148b28ae280afa0b2eb33aa868ff64e3e87502cd63a8ca881a4380c0b4e07e8
Oct 30 19:49:38.284269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:38.284230    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb" event={"ID":"acce9ca9-bca5-4932-9ceb-35cc4bd63055","Type":"ContainerStarted","Data":"5148b28ae280afa0b2eb33aa868ff64e3e87502cd63a8ca881a4380c0b4e07e8"}
Oct 30 19:49:39.294516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.294440    4519 generic.go:356] "Generic (PLEG): container finished" podID="acce9ca9-bca5-4932-9ceb-35cc4bd63055" containerID="22d95e2369b05a4eae1b67d501d0b62e0bf0d58769a858a4997ced09b6b3cec1" exitCode=0
Oct 30 19:49:39.294979 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.294649    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb" event={"ID":"acce9ca9-bca5-4932-9ceb-35cc4bd63055","Type":"ContainerDied","Data":"22d95e2369b05a4eae1b67d501d0b62e0bf0d58769a858a4997ced09b6b3cec1"}
Oct 30 19:49:39.312386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.312274    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"]
Oct 30 19:49:39.316618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.316592    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.334957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.334926    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"]
Oct 30 19:49:39.461016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.460975    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5rp\" (UniqueName: \"kubernetes.io/projected/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-kube-api-access-tm5rp\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.461238 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.461205    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.461329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.461300    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.562740 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.562625    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.562740 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.562697    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.562990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.562821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5rp\" (UniqueName: \"kubernetes.io/projected/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-kube-api-access-tm5rp\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.563674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.563635    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.565984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.565958    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.582679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.582644    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5rp\" (UniqueName: \"kubernetes.io/projected/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-kube-api-access-tm5rp\") pod \"maas-api-revoke-sub-27f800c7-5f6qt\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.630512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.630449    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:39.937376 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:39.937343    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"]
Oct 30 19:49:40.304098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.304003    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt" event={"ID":"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c","Type":"ContainerStarted","Data":"4890a490207d388f11b54d0e68e236d6fa45920871c47c6e99b875caf3879aac"}
Oct 30 19:49:40.473964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.473927    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:40.577362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.577251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-xks-ca\") pod \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") "
Oct 30 19:49:40.577362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.577316    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acce9ca9-bca5-4932-9ceb-35cc4bd63055-kube-api-access-8p4cc\" (UniqueName: \"kubernetes.io/projected/acce9ca9-bca5-4932-9ceb-35cc4bd63055-kube-api-access-8p4cc\") pod \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") "
Oct 30 19:49:40.577362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.577360    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-openshift-ca\") pod \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\" (UID: \"acce9ca9-bca5-4932-9ceb-35cc4bd63055\") "
Oct 30 19:49:40.578603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.578560    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-openshift-ca" pod "acce9ca9-bca5-4932-9ceb-35cc4bd63055" (UID: "acce9ca9-bca5-4932-9ceb-35cc4bd63055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:40.581484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.581436    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acce9ca9-bca5-4932-9ceb-35cc4bd63055-kube-api-access-8p4cc" pod "acce9ca9-bca5-4932-9ceb-35cc4bd63055" (UID: "acce9ca9-bca5-4932-9ceb-35cc4bd63055"). InnerVolumeSpecName "kube-api-access-8p4cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:40.581873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.581832    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-xks-ca" pod "acce9ca9-bca5-4932-9ceb-35cc4bd63055" (UID: "acce9ca9-bca5-4932-9ceb-35cc4bd63055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:40.690421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.686400    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:40.690421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.686457    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4cc\" (UniqueName: \"kubernetes.io/projected/acce9ca9-bca5-4932-9ceb-35cc4bd63055-kube-api-access-8p4cc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:40.690421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:40.686487    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acce9ca9-bca5-4932-9ceb-35cc4bd63055-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:41.314841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:41.314797    4519 generic.go:356] "Generic (PLEG): container finished" podID="d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" containerID="338c090ce2b767ea829458aec5cbc0bb90ab1ad7d1402ac7553629766dcbc01a" exitCode=0
Oct 30 19:49:41.315357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:41.314886    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt" event={"ID":"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c","Type":"ContainerDied","Data":"338c090ce2b767ea829458aec5cbc0bb90ab1ad7d1402ac7553629766dcbc01a"}
Oct 30 19:49:41.316594 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:41.316569    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb" event={"ID":"acce9ca9-bca5-4932-9ceb-35cc4bd63055","Type":"ContainerDied","Data":"5148b28ae280afa0b2eb33aa868ff64e3e87502cd63a8ca881a4380c0b4e07e8"}
Oct 30 19:49:41.316730 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:41.316610    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5148b28ae280afa0b2eb33aa868ff64e3e87502cd63a8ca881a4380c0b4e07e8"
Oct 30 19:49:41.316730 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:41.316620    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"
Oct 30 19:49:42.095592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.095542    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"]
Oct 30 19:49:42.100490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.096832    4519 cpu_manager.go:414] "containerMap: removing container" podUID="acce9ca9-bca5-4932-9ceb-35cc4bd63055" containerName="revoke-keys"
Oct 30 19:49:42.100490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.096864    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acce9ca9-bca5-4932-9ceb-35cc4bd63055" containerName="revoke-keys"
Oct 30 19:49:42.100490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.097171    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="acce9ca9-bca5-4932-9ceb-35cc4bd63055" containerName="revoke-keys"
Oct 30 19:49:42.164525 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.159844    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.176342 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.176291    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"]
Oct 30 19:49:42.405510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.380535    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"]
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.487781    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.487865    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.488001    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.488037    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9pgt\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-kube-api-access-w9pgt\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.488182    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.488235    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.488262    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.488286    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591e7d80-491c-4c58-802a-fc833b7ea25c-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.496489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.488338    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.514493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.504454    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"]
Oct 30 19:49:42.530677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.530634    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.537787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.537739    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"]
Oct 30 19:49:42.537958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.537801    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"]
Oct 30 19:49:42.537958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.537911    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.589084    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.589147    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9pgt\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-kube-api-access-w9pgt\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.589253    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.589294    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.589764    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.590290    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.590742    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.593400    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591e7d80-491c-4c58-802a-fc833b7ea25c-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.593816    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591e7d80-491c-4c58-802a-fc833b7ea25c-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.593884    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.593916    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.593958    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.594246    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.594413    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.597408    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.606751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.598070    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.636404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.636365    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.643692 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.643647    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9pgt\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-kube-api-access-w9pgt\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:42.697174 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697075    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7bcafa-0551-4059-87db-5b466c2e914a-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.697174 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697145    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.697430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697252    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxjh\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-kube-api-access-kbxjh\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.697430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697334    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.697430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697366    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.697607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697452    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.697607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697518    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.697742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697696    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.697799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697745    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.697799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.697897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697805    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.697897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697837    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.697897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697889    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.698039 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.697934    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.698039 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.698007    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzszb\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-kube-api-access-nzszb\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.698145 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.698051    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.698145 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.698088    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.698145 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.698117    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.744532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.744500    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:42.799827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.799784    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-openshift-ca\") pod \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") "
Oct 30 19:49:42.800032 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.799864    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-kube-api-access-tm5rp\" (UniqueName: \"kubernetes.io/projected/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-kube-api-access-tm5rp\") pod \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") "
Oct 30 19:49:42.800032 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.799970    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-xks-ca\") pod \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\" (UID: \"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c\") "
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800193    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800263    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800291    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800315    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800352    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800385    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800437    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800590    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzszb\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-kube-api-access-nzszb\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800633    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800669    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800695    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800730    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7bcafa-0551-4059-87db-5b466c2e914a-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800736    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800766    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800791    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxjh\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-kube-api-access-kbxjh\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800838    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.800882    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.801063    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.801240    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.801605    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.801841    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.802587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.802076    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.803925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.802713    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7bcafa-0551-4059-87db-5b466c2e914a-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.803925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.802795    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.803925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.802959    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-xks-ca" pod "d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" (UID: "d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:42.803925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.803260    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.803925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.803697    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-openshift-ca" pod "d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" (UID: "d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:42.803925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.803731    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.811322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.805731    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.811322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.806922    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-kube-api-access-tm5rp" pod "d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" (UID: "d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c"). InnerVolumeSpecName "kube-api-access-tm5rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:42.811322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.807309    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.811322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.807636    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.811322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.809186    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.821932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.821879    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.822115 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.822090    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxjh\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-kube-api-access-kbxjh\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.822190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.822164    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.823198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.823173    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzszb\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-kube-api-access-nzszb\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.886872 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.886777    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:42.893446 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.893411    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:42.902451 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.902422    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:42.902580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.902454    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm5rp\" (UniqueName: \"kubernetes.io/projected/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-kube-api-access-tm5rp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:42.902580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.902483    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:49:42.941279 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:42.941235    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:43.229038 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:43.229004    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f33475_5927_4d76_a197_4a5d0fb1aa94.slice/crio-4b4d0bca15850c72c23b15ce6d115d1d7c452d75b4e8de6b24d18d01ae0958da WatchSource:0}: Error finding container 4b4d0bca15850c72c23b15ce6d115d1d7c452d75b4e8de6b24d18d01ae0958da: Status 404 returned error can't find the container with id 4b4d0bca15850c72c23b15ce6d115d1d7c452d75b4e8de6b24d18d01ae0958da
Oct 30 19:49:43.231942 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.231905    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.232067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.231953    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"]
Oct 30 19:49:43.232067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.231976    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.232067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.232008    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.236723 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.236690    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"]
Oct 30 19:49:43.242879 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:43.242841    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7bcafa_0551_4059_87db_5b466c2e914a.slice/crio-3fadc7e91a4f84ccb4755d4846387a15b723ae1d9fa7d5ca8478d3878bd1e722 WatchSource:0}: Error finding container 3fadc7e91a4f84ccb4755d4846387a15b723ae1d9fa7d5ca8478d3878bd1e722: Status 404 returned error can't find the container with id 3fadc7e91a4f84ccb4755d4846387a15b723ae1d9fa7d5ca8478d3878bd1e722
Oct 30 19:49:43.245598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.245558    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.245711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.245649    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.245711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.245694    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.282539 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.282498    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"]
Oct 30 19:49:43.286236 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:49:43.286199    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591e7d80_491c_4c58_802a_fc833b7ea25c.slice/crio-41cd85a1aed10f8f58aed5b864757834b521598b0c5bff035c211f7f6dffe8ea WatchSource:0}: Error finding container 41cd85a1aed10f8f58aed5b864757834b521598b0c5bff035c211f7f6dffe8ea: Status 404 returned error can't find the container with id 41cd85a1aed10f8f58aed5b864757834b521598b0c5bff035c211f7f6dffe8ea
Oct 30 19:49:43.289342 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.289296    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.289492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.289384    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.289492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.289427    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:43.363071 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.361968    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" event={"ID":"9e7bcafa-0551-4059-87db-5b466c2e914a","Type":"ContainerStarted","Data":"838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e"}
Oct 30 19:49:43.363071 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.362035    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" event={"ID":"9e7bcafa-0551-4059-87db-5b466c2e914a","Type":"ContainerStarted","Data":"3fadc7e91a4f84ccb4755d4846387a15b723ae1d9fa7d5ca8478d3878bd1e722"}
Oct 30 19:49:43.373580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.365444    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt" event={"ID":"d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c","Type":"ContainerDied","Data":"4890a490207d388f11b54d0e68e236d6fa45920871c47c6e99b875caf3879aac"}
Oct 30 19:49:43.373580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.369878    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4890a490207d388f11b54d0e68e236d6fa45920871c47c6e99b875caf3879aac"
Oct 30 19:49:43.373580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.369987    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"
Oct 30 19:49:43.377720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.376777    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" event={"ID":"f0f33475-5927-4d76-a197-4a5d0fb1aa94","Type":"ContainerStarted","Data":"9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e"}
Oct 30 19:49:43.377720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.376829    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" event={"ID":"f0f33475-5927-4d76-a197-4a5d0fb1aa94","Type":"ContainerStarted","Data":"4b4d0bca15850c72c23b15ce6d115d1d7c452d75b4e8de6b24d18d01ae0958da"}
Oct 30 19:49:43.380824 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.380741    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" event={"ID":"591e7d80-491c-4c58-802a-fc833b7ea25c","Type":"ContainerStarted","Data":"41cd85a1aed10f8f58aed5b864757834b521598b0c5bff035c211f7f6dffe8ea"}
Oct 30 19:49:43.417139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.417063    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podStartSLOduration=1.417041726 podStartE2EDuration="1.417041726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:43.413665082 +0000 UTC m=+907.914368822" watchObservedRunningTime="2026-10-30 19:49:43.417041726 +0000 UTC m=+907.917745452"
Oct 30 19:49:43.481646 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.480157    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podStartSLOduration=1.4801324359999999 podStartE2EDuration="1.480132436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:43.47324112 +0000 UTC m=+907.973944836" watchObservedRunningTime="2026-10-30 19:49:43.480132436 +0000 UTC m=+907.980836160"
Oct 30 19:49:43.887448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.887351    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:43.894564 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:43.894522    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:44.430490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.426665    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" event={"ID":"591e7d80-491c-4c58-802a-fc833b7ea25c","Type":"ContainerStarted","Data":"14c935898a5f231cda9d90fb94c96539039b10dc3380e6b26342512a85e00bdb"}
Oct 30 19:49:44.535295 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.535220    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podStartSLOduration=2.535187116 podStartE2EDuration="2.535187116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:44.485989008 +0000 UTC m=+908.986692733" watchObservedRunningTime="2026-10-30 19:49:44.535187116 +0000 UTC m=+909.035890841"
Oct 30 19:49:44.940492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.934851    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:49:44.940492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.934930    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:49:44.940492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.935292    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:49:44.940492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.935327    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:49:44.952614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.952574    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:44.952945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.952707    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:49:44.953109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:44.953084    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:49:45.887958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:45.887914    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:49:45.888607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:45.888574    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:49:45.897977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:45.897939    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:49:45.898228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:45.898201    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:49:45.943023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:45.942970    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:49:45.943207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:45.943039    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:49:46.888072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:46.888026    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:49:46.888771 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:46.888726    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:49:46.896036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:46.896000    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:49:46.896353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:46.896309    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:49:46.942950 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:46.942913    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:49:46.943223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:46.943183    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:49:47.907494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:47.888912    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:49:47.907494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:47.888982    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:49:47.907494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:47.896935    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:49:47.907494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:47.897000    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:49:47.943502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:47.942104    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:49:47.943502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:47.942156    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:49:48.888055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:48.888016    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:49:48.888343 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:48.888303    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:49:48.921493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:48.915512    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:49:48.921493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:48.915588    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:49:48.952688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:48.952647    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:49:48.952941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:48.952915    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:49:49.890058 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:49.890008    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:49:49.890408 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:49.890367    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:49:49.895034 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:49.894986    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:49:49.895333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:49.895301    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:49:49.942063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:49.942018    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:49:49.942586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:49.942087    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:49:51.139310 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.139109    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:51.140672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.140643    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:51.143167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.143123    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:49:51.226556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.226521    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:51.227947 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.227921    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:51.230813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.230790    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:49:51.351581 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.351430    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:51.353079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.352919    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:49:51.355380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:49:51.355341    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:50:02.233385 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:50:02.233338    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:50:06.670409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.670368    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"]
Oct 30 19:50:06.671428 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.671399    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" containerName="revoke-keys"
Oct 30 19:50:06.671428 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.671428    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" containerName="revoke-keys"
Oct 30 19:50:06.671650 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.671600    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" containerName="revoke-keys"
Oct 30 19:50:06.677438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.677402    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.700031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.699996    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"]
Oct 30 19:50:06.700790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.700757    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"openshift-service-ca.crt\""
Oct 30 19:50:06.700948 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.700877    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"kube-root-ca.crt\""
Oct 30 19:50:06.701054 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.700977    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"distinct-2-w0-5db436-kserve-self-signed-certs\""
Oct 30 19:50:06.707337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.707299    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"default-dockercfg-jb4r7\""
Oct 30 19:50:06.775961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.775914    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6db\" (UniqueName: \"kubernetes.io/projected/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-kube-api-access-7v6db\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.776179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.776150    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tls-certs\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.776234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.776203    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tmp-dir\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.776338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.776317    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-dshm\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.776395 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.776380    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-home\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.776533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.776514    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-model-cache\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.880672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.880626    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tls-certs\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.881411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.881382    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tmp-dir\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.881644 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.881624    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-dshm\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.881801 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.881781    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-home\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.881976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.881960    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-model-cache\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.882239 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.882219    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6db\" (UniqueName: \"kubernetes.io/projected/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-kube-api-access-7v6db\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.883168 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.883143    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-home\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.883322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.883172    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-model-cache\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.885747 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.883412    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tmp-dir\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.898893 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.898848    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-dshm\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.899295 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.899262    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tls-certs\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:06.925551 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:06.925167    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6db\" (UniqueName: \"kubernetes.io/projected/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-kube-api-access-7v6db\") pod \"distinct-2-w0-5db436-kserve-859b87564d-cmjft\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:07.004414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:07.004378    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:07.342875 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:07.342837    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda39f4406_705a_4fc3_9f5d_46c2cfd750f8.slice/crio-b112e5a3471e6286ee547248312cd50cbb3d3a7ea59757bf4468a3a382d3c66b WatchSource:0}: Error finding container b112e5a3471e6286ee547248312cd50cbb3d3a7ea59757bf4468a3a382d3c66b: Status 404 returned error can't find the container with id b112e5a3471e6286ee547248312cd50cbb3d3a7ea59757bf4468a3a382d3c66b
Oct 30 19:50:07.345221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:07.345195    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"]
Oct 30 19:50:07.757503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:07.757418    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" event={"ID":"a39f4406-705a-4fc3-9f5d-46c2cfd750f8","Type":"ContainerStarted","Data":"371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173"}
Oct 30 19:50:07.757940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:07.757534    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" event={"ID":"a39f4406-705a-4fc3-9f5d-46c2cfd750f8","Type":"ContainerStarted","Data":"b112e5a3471e6286ee547248312cd50cbb3d3a7ea59757bf4468a3a382d3c66b"}
Oct 30 19:50:07.798574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:07.798514    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podStartSLOduration=1.79845567 podStartE2EDuration="1.79845567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:07.796552411 +0000 UTC m=+932.297256137" watchObservedRunningTime="2026-10-30 19:50:07.79845567 +0000 UTC m=+932.299159393"
Oct 30 19:50:08.004921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:08.004876    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:11.816292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.816250    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"]
Oct 30 19:50:11.821647 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.821617    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.855186 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.855143    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"]
Oct 30 19:50:11.965725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.965677    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.965725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.965726    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l75s\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-kube-api-access-5l75s\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.965957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.965838    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.965957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.965892    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.966029 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.965970    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.966064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.966028    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.966100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.966063    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.966100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.966090    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:11.966194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:11.966181    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.069685 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069541    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.069685 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069612    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.069939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069693    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.069939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069726    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.069939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069773    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.069939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069871    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.069939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069925    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.070168 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.069970    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l75s\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-kube-api-access-5l75s\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.070168 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.070044    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.070972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.070938    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.071280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.071255    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.076112 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.074140    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.076112 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.074415    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.076112 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.075407    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.076112 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.075715    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.079682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.079644    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.095927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.095890    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.099155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.099119    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l75s\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-kube-api-access-5l75s\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.137226 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.137187    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:12.480820 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:12.480781    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cdc4a2_8d8c_4190_9e6e_045499cc99f4.slice/crio-a8a92faa797959d68809bb9c7d2021d7a32fa19a0b7be5c67af9eef8d75dcfed WatchSource:0}: Error finding container a8a92faa797959d68809bb9c7d2021d7a32fa19a0b7be5c67af9eef8d75dcfed: Status 404 returned error can't find the container with id a8a92faa797959d68809bb9c7d2021d7a32fa19a0b7be5c67af9eef8d75dcfed
Oct 30 19:50:12.481874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.481850    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"]
Oct 30 19:50:12.483303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.483270    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.483385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.483336    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.483385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.483364    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.606814 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.606781    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"]
Oct 30 19:50:12.611264 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.611236    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.640139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.640090    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"]
Oct 30 19:50:12.678489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678028    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.678489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678084    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.678489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678144    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f9d75c-5271-41f1-9777-6216ad94edf8-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.678489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678243    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.678489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678317    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8tg\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-kube-api-access-8s8tg\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.678489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678372    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.678489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678407    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.679059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678534    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.679059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.678588    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780346 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780249    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f9d75c-5271-41f1-9777-6216ad94edf8-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780346 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780334    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780379    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8tg\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-kube-api-access-8s8tg\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780423    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780686    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780732    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780792    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.780825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.780816    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.781246 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.781220    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.781302 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.781249    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.781302 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.781264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.781522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.781504    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.781623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.781546    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f9d75c-5271-41f1-9777-6216ad94edf8-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.783788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.783752    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.783899 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.783878    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.814097 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.814058    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.814311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.814195    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8tg\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-kube-api-access-8s8tg\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.815666 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.815612    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" event={"ID":"53cdc4a2-8d8c-4190-9e6e-045499cc99f4","Type":"ContainerStarted","Data":"bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934"}
Oct 30 19:50:12.815816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.815690    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" event={"ID":"53cdc4a2-8d8c-4190-9e6e-045499cc99f4","Type":"ContainerStarted","Data":"a8a92faa797959d68809bb9c7d2021d7a32fa19a0b7be5c67af9eef8d75dcfed"}
Oct 30 19:50:12.866190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.866128    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podStartSLOduration=1.8661107700000001 podStartE2EDuration="1.86611077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:12.860937541 +0000 UTC m=+937.361641265" watchObservedRunningTime="2026-10-30 19:50:12.86611077 +0000 UTC m=+937.366814582"
Oct 30 19:50:12.901590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.901547    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"]
Oct 30 19:50:12.907162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.907133    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.928615 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.928582    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:12.952694 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.952658    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"]
Oct 30 19:50:12.984095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.983724    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6w9\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-kube-api-access-8g6w9\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.984095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.983794    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.984095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.983860    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.984437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.984336    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.984609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.984535    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.984806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.984781    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.984884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.984856    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.984982 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.984963    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:12.985082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:12.985066    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098385    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6w9\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-kube-api-access-8g6w9\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098450    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098521    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098547    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098622    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098752    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098779    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098841    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.098896    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.099422    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.100689    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.107030    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.107850    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.108059    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.108701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.108269    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.110454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.110402    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.123542 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.123349    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6w9\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-kube-api-access-8g6w9\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.123542 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.123407    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.138161 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.138121    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:13.139673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.139636    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:13.139859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.139720    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:13.229504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.229452    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:13.284919 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:13.284880    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f9d75c_5271_41f1_9777_6216ad94edf8.slice/crio-385ed133d071f075fb26f6d472e69be0307257797fd541e2817bef41e65dbb75 WatchSource:0}: Error finding container 385ed133d071f075fb26f6d472e69be0307257797fd541e2817bef41e65dbb75: Status 404 returned error can't find the container with id 385ed133d071f075fb26f6d472e69be0307257797fd541e2817bef41e65dbb75
Oct 30 19:50:13.285382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.285355    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"]
Oct 30 19:50:13.287651 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.287618    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:13.287740 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.287700    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:13.287740 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.287731    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:13.601966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.601928    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"]
Oct 30 19:50:13.611929 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.611890    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:13.612073 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.611974    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:13.612073 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.612019    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:13.828121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.828078    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" event={"ID":"30f9d75c-5271-41f1-9777-6216ad94edf8","Type":"ContainerStarted","Data":"db5729905b2d45222c6aeac75c503553764aea44d6dd778bb2924466cc4fbade"}
Oct 30 19:50:13.828334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.828147    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" event={"ID":"30f9d75c-5271-41f1-9777-6216ad94edf8","Type":"ContainerStarted","Data":"385ed133d071f075fb26f6d472e69be0307257797fd541e2817bef41e65dbb75"}
Oct 30 19:50:13.830524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.830336    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" event={"ID":"ea0509ee-c53f-4ea0-a1f4-d50be64daf27","Type":"ContainerStarted","Data":"08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded"}
Oct 30 19:50:13.830524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.830397    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" event={"ID":"ea0509ee-c53f-4ea0-a1f4-d50be64daf27","Type":"ContainerStarted","Data":"560703f8054edc2188b3afddb9c86bb6a6a3493dd35598587922cad8f4ff18db"}
Oct 30 19:50:13.888008 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.887935    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podStartSLOduration=1.887909826 podStartE2EDuration="1.887909826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:13.880021726 +0000 UTC m=+938.380725452" watchObservedRunningTime="2026-10-30 19:50:13.887909826 +0000 UTC m=+938.388613555"
Oct 30 19:50:13.929528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.929482    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:13.935289 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:13.935185    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podStartSLOduration=1.935162326 podStartE2EDuration="1.935162326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:13.927632733 +0000 UTC m=+938.428336472" watchObservedRunningTime="2026-10-30 19:50:13.935162326 +0000 UTC m=+938.435866050"
Oct 30 19:50:14.138401 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:14.138355    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.138598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:14.138423    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:14.230779 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:14.230678    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:14.232126 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:14.232083    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.232276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:14.232164    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:14.950494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:14.939673    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.950494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:14.939760    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:50:15.140929 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:15.140888    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.141215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:15.141185    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:15.248551 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:15.244619    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.248551 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:15.244689    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:15.936661 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:15.936623    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.936979 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:15.936949    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:50:16.141967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:16.141927    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.142684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:16.142650    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:16.236437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:16.236339    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.236770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:16.236740    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:16.938487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:16.938025    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.938487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:16.938091    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:50:17.010532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.010447    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:17.139347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.139030    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.139347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.139098    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:17.147490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.144090    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:17.240392 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.237421    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.240392 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.237503    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:17.937495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.932630    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.937495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:17.932692    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:50:18.037985 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.037955    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:18.140487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.139913    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:18.140487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.139980    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:18.251564 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.230300    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:18.261495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.255549    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:18.757741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.757618    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:18.932925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.932886    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:50:18.933194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.933168    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:50:18.977799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:18.977667    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:19.142553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:19.141504    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:19.142553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:19.141573    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:19.235491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:19.234973    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:19.235491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:19.235040    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:20.006710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.006667    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:50:20.007292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.007246    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:50:20.139604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.137753    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:50:20.139604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.137814    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:50:20.233511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.229912    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:20.233511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.229978    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:20.929365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.929323    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:50:20.929579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:20.929388    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:50:21.142986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.142948    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:21.143537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.143394    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:21.144392 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.144370    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 19:50:21.230589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.230460    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:50:21.230589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.230550    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:50:21.935747 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.935714    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:21.936247 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.936214    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:21.937066 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:21.937039    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:50:22.235375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:22.235291    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:22.235905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:22.235878    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:22.236935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:22.236913    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 19:50:27.086964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.086920    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"]
Oct 30 19:50:27.096905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.096863    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.113127 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.113095    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"]
Oct 30 19:50:27.121649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.121613    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"openshift-service-ca.crt\""
Oct 30 19:50:27.128509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.128459    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"default-dockercfg-7v28s\""
Oct 30 19:50:27.128509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.128460    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"kube-root-ca.crt\""
Oct 30 19:50:27.128743 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.128536    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"distinct-2-w1-109cfe-kserve-self-signed-certs\""
Oct 30 19:50:27.219299 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.219249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlkr\" (UniqueName: \"kubernetes.io/projected/8232473a-8c1a-4f51-80d4-578eee5bc87c-kube-api-access-jdlkr\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.219517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.219380    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8232473a-8c1a-4f51-80d4-578eee5bc87c-tls-certs\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.219517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.219436    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-tmp-dir\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.219517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.219492    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-home\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.219630 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.219565    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-dshm\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.219630 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.219585    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-model-cache\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.325489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.324337    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlkr\" (UniqueName: \"kubernetes.io/projected/8232473a-8c1a-4f51-80d4-578eee5bc87c-kube-api-access-jdlkr\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.325489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.324438    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8232473a-8c1a-4f51-80d4-578eee5bc87c-tls-certs\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.325489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.324534    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-tmp-dir\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.325489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.324586    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-home\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.325489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.324620    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-dshm\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.325489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.324644    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-model-cache\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.325489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.325199    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-model-cache\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.332490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.326645    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-tmp-dir\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.332490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.326953    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-home\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.339560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.337664    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8232473a-8c1a-4f51-80d4-578eee5bc87c-tls-certs\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.362491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.353219    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-dshm\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.374793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.374759    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlkr\" (UniqueName: \"kubernetes.io/projected/8232473a-8c1a-4f51-80d4-578eee5bc87c-kube-api-access-jdlkr\") pod \"distinct-2-w1-109cfe-kserve-68684f46b7-px7kk\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.416835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.416504    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:27.753963 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:27.753918    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8232473a_8c1a_4f51_80d4_578eee5bc87c.slice/crio-bd1d4b957666b3bc9db3e19f42c742a78696d82f423010e18065e095709f2d69 WatchSource:0}: Error finding container bd1d4b957666b3bc9db3e19f42c742a78696d82f423010e18065e095709f2d69: Status 404 returned error can't find the container with id bd1d4b957666b3bc9db3e19f42c742a78696d82f423010e18065e095709f2d69
Oct 30 19:50:27.756041 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:27.756006    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"]
Oct 30 19:50:28.208568 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:28.208529    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" event={"ID":"8232473a-8c1a-4f51-80d4-578eee5bc87c","Type":"ContainerStarted","Data":"3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44"}
Oct 30 19:50:28.208996 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:28.208576    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" event={"ID":"8232473a-8c1a-4f51-80d4-578eee5bc87c","Type":"ContainerStarted","Data":"bd1d4b957666b3bc9db3e19f42c742a78696d82f423010e18065e095709f2d69"}
Oct 30 19:50:28.256354 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:28.256280    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podStartSLOduration=1.256256869 podStartE2EDuration="1.256256869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:28.250582348 +0000 UTC m=+952.751286073" watchObservedRunningTime="2026-10-30 19:50:28.256256869 +0000 UTC m=+952.756960594"
Oct 30 19:50:28.417080 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:28.417042    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:32.230505 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:50:32.230441    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:50:37.417716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:37.417671    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:37.436002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:37.435964    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:38.305048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:38.305011    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:42.048945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.048902    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"]
Oct 30 19:50:42.054485 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.054445    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.105424 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.105386    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"]
Oct 30 19:50:42.138218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138175    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138415 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138318    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbqs\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-kube-api-access-qfbqs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138415 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138371    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138542 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138434    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138534    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138570    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138599    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138690    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.138749 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.138732    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.239910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.239870    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.239922    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.239966    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240088    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240259 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240181    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbqs\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-kube-api-access-qfbqs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240259 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240213    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240322    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240424 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240360    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240424 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240389    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240507    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240708 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240659    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240793    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240805    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.240851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.240842    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.242549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.242523    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.242874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.242856    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.273034 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.272990    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbqs\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-kube-api-access-qfbqs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.273278 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.273251    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"]
Oct 30 19:50:42.273904 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.273881    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.278975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.278946    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.309274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.309242    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"]
Oct 30 19:50:42.379102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.379054    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:42.442780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.442750    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.442797    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951a800-4431-4b8a-b6b7-2b400d641c2f-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.442830    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.442878    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.442930    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.442972    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.443003    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.443082    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7xj\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-kube-api-access-jl7xj\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.445311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.443288    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.524100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.524059    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"]
Oct 30 19:50:42.529425 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.529394    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.544314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544281    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7xj\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-kube-api-access-jl7xj\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.544454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544407    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.544454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544449    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.544586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951a800-4431-4b8a-b6b7-2b400d641c2f-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.544586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544532    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.544586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544560    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.544780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544592    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.545208 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.545182    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.545392 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.545369    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.545452 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.544663    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.545550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.545444    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.545550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.545518    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.545721 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.545702    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.546060 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.546036    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951a800-4431-4b8a-b6b7-2b400d641c2f-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.552856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.552766    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5db436-openshift-default-dockercfg-n7492\""
Oct 30 19:50:42.565204 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.565060    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"]
Oct 30 19:50:42.585955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.585917    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.586190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.586157    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.586635 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.586613    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.586762 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.586631    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7xj\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-kube-api-access-jl7xj\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.611104 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.611059    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:42.646692 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.646636    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958xx\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-kube-api-access-958xx\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.646887 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.646725    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.646887 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.646765    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.647057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.646941    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.647057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.646984    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.647057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.647008    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.647057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.647050    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.647285 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.647105    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.647285 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.647143    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.746276 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:42.746239    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f478e2_4b31_4dba_99ff_b96cecc9b1d4.slice/crio-97f0237f647aa485ea2e1e2a8e1f945f67d2cebb8ec344d2c3d7545141272554 WatchSource:0}: Error finding container 97f0237f647aa485ea2e1e2a8e1f945f67d2cebb8ec344d2c3d7545141272554: Status 404 returned error can't find the container with id 97f0237f647aa485ea2e1e2a8e1f945f67d2cebb8ec344d2c3d7545141272554
Oct 30 19:50:42.748096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748059    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-958xx\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-kube-api-access-958xx\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748187 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748148    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748190    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748300 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748285    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748335 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748316    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748378 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748340    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748459    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.748536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.748517    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.749597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.749565    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"]
Oct 30 19:50:42.749796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.749768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.750043 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.750018    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.750898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.750869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.751154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.751133    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.754600 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.753893    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.754600 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.754030    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:42.754600 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.754123    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:42.754600 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.754170    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:42.754600 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.754359    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.773354 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.773319    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.774023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.773998    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958xx\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-kube-api-access-958xx\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.783067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.783032    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.846133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.846100    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:42.995443 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:42.995336    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"]
Oct 30 19:50:43.010749 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:43.010692    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4951a800_4431_4b8a_b6b7_2b400d641c2f.slice/crio-9baefddff8c4aee45a51ac2f9e1bc9623c561274afbdb5de11eeec2233209566 WatchSource:0}: Error finding container 9baefddff8c4aee45a51ac2f9e1bc9623c561274afbdb5de11eeec2233209566: Status 404 returned error can't find the container with id 9baefddff8c4aee45a51ac2f9e1bc9623c561274afbdb5de11eeec2233209566
Oct 30 19:50:43.015564 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.015429    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:43.015709 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.015546    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:43.015709 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.015656    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:43.208071 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.208032    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"]
Oct 30 19:50:43.211249 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:43.211215    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1633beb3_9fe0_49bb_89f1_f5cb464e4dec.slice/crio-b693225f2c4e02a4d91b24c24fa5243f0705a4478f60525e96f52b9aa1a22019 WatchSource:0}: Error finding container b693225f2c4e02a4d91b24c24fa5243f0705a4478f60525e96f52b9aa1a22019: Status 404 returned error can't find the container with id b693225f2c4e02a4d91b24c24fa5243f0705a4478f60525e96f52b9aa1a22019
Oct 30 19:50:43.214102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.214066    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:43.214214 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.214172    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:43.214255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.214210    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:43.339215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.339114    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" event={"ID":"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4","Type":"ContainerStarted","Data":"d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815"}
Oct 30 19:50:43.339215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.339189    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" event={"ID":"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4","Type":"ContainerStarted","Data":"97f0237f647aa485ea2e1e2a8e1f945f67d2cebb8ec344d2c3d7545141272554"}
Oct 30 19:50:43.343568 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.343513    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" event={"ID":"1633beb3-9fe0-49bb-89f1-f5cb464e4dec","Type":"ContainerStarted","Data":"b693225f2c4e02a4d91b24c24fa5243f0705a4478f60525e96f52b9aa1a22019"}
Oct 30 19:50:43.345526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.345498    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" event={"ID":"4951a800-4431-4b8a-b6b7-2b400d641c2f","Type":"ContainerStarted","Data":"717cac305f3d8aa180c449d9f23564b1fcda0013912d27c8fb399a14f56d7550"}
Oct 30 19:50:43.345642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.345556    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" event={"ID":"4951a800-4431-4b8a-b6b7-2b400d641c2f","Type":"ContainerStarted","Data":"9baefddff8c4aee45a51ac2f9e1bc9623c561274afbdb5de11eeec2233209566"}
Oct 30 19:50:43.381057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.380934    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:43.381291 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.381258    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:43.381358 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.381312    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:43.400257 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.400187    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podStartSLOduration=2.400165859 podStartE2EDuration="2.400165859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:43.399732747 +0000 UTC m=+967.900436476" watchObservedRunningTime="2026-10-30 19:50:43.400165859 +0000 UTC m=+967.900869583"
Oct 30 19:50:43.614079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:43.613996    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:44.382491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.380083    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:44.382491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.380149    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:44.402662 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.402620    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" event={"ID":"1633beb3-9fe0-49bb-89f1-f5cb464e4dec","Type":"ContainerStarted","Data":"8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c"}
Oct 30 19:50:44.491283 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.491214    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podStartSLOduration=2.491114119 podStartE2EDuration="2.491114119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:43.459830288 +0000 UTC m=+967.960534029" watchObservedRunningTime="2026-10-30 19:50:44.491114119 +0000 UTC m=+968.991817843"
Oct 30 19:50:44.615297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.615251    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:44.616082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.615951    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:44.850970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.850937    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:44.855153 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.855102    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:44.855478 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:44.855419    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:45.381610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:45.381545    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:45.381841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:45.381609    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:45.645298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:45.644632    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:45.645298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:45.644723    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:45.866492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:45.862816    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:45.866492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:45.862918    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:46.387493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:46.380503    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:46.387493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:46.380566    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:46.613493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:46.613009    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:46.613493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:46.613074    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:46.858492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:46.855312    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:46.858492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:46.855392    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:46.858492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:46.855866    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:47.383488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.382935    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:47.383488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.382999    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:47.628489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.624888    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:47.628489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.624956    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:47.852489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.851907    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:47.852489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.851972    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:47.860657 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.860596    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:47.861793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.861769    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:50:47.881287 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.880867    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:47.948201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.948128    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podStartSLOduration=5.948106277 podStartE2EDuration="5.948106277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:44.487986733 +0000 UTC m=+968.988690458" watchObservedRunningTime="2026-10-30 19:50:47.948106277 +0000 UTC m=+972.448810002"
Oct 30 19:50:47.996777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:47.996632    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:48.028409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.028086    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:48.054521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.048633    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:48.277510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.268702    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Oct 30 19:50:48.349601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.348521    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:48.394492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.391021    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:48.394492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.391104    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:48.398488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.397627    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:48.467491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.466612    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:48.615553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.614969    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:48.615553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.615048    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:48.629771 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.627587    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:50:48.850493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.849894    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:48.850493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.849962    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:48.869553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.861571    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:50:48.889489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.882564    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:48.889489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.883029    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:50:48.942493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:48.937775    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:50:49.003491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.002578    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:49.003491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.003437    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:49.059493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.048963    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:49.086493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.077509    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:50:49.099102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.088705    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:49.350805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.350576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:50:49.351352 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.351323    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:49.395593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.383207    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:49.395593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.383272    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:49.688491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.674755    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:49.688491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.675438    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:50:49.765492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.759054    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:49.765492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.759126    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:49.868563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.868509    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:49.872556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.872516    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:49.873146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:49.873115    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:50.024853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.024733    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:50.028779 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.028727    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:50.119490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.115571    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:50:50.177729 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.177677    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:50.316736 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.313705    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:50:50.316736 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.313801    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:50:50.316736 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.313864    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:50:50.359489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.355328    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:50.499493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.479528    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:50.499493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.479598    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:50.523637 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.523575    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 19:50:50.631412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.631303    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:50.631783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.631737    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:50.697656 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.697573    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:50.763778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.763729    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:50.849122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.849081    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:50.849408 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.849376    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:50.885704 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.885577    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:50.935348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:50.934853    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:51.048492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.025578    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:50:51.074381 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.051317    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:50:51.104806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.104756    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:51.295010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.268994    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:50:51.295010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.294569    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:50:51.383502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.383435    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:51.490528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.454514    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:51.553869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.553728    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:51.665503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.637674    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:51.665503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.637751    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:51.665503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.641185    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:51.665503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.641243    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:51.722887 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.722038    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:51.784667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.784616    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:51.888558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.868667    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:51.888558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.868747    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:51.888558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.871597    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:50:51.904495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.889339    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:51.955494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.948617    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:50:51.955902 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:51.955868    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:50:52.015494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.012736    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"]
Oct 30 19:50:52.074494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.030619    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.074869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.074827    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:52.103604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.103552    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.125492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.076225    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.274509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.270549    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.300624 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.300574    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:50:52.352493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.346563    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.415805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.393795    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"]
Oct 30 19:50:52.416193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.404885    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:52.416442 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.416396    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:52.416981 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.416940    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:50:52.417620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.417593    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:50:52.465495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.421111    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.465495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.432628    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:50:52.489493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.488739    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:52.489493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.488816    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.549548 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.544665    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zr7m\" (UniqueName: \"kubernetes.io/projected/b1c53bd7-b699-4371-b434-d7882000083a-kube-api-access-2zr7m\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.549548 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.544739    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.549548 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.544898    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.636265 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.634646    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:52.636265 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.634723    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:52.675671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.675615    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:52.675967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.675937    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.692493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.692388    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zr7m\" (UniqueName: \"kubernetes.io/projected/b1c53bd7-b699-4371-b434-d7882000083a-kube-api-access-2zr7m\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.692493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.692487    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.692765 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.692641    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.701498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.693733    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.751503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.742862    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:50:52.807879 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.807770    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:50:52.880492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.856048    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:52.882144 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.882102    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:52.905884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.872853    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.906440 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.889985    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:52.925493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.922265    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zr7m\" (UniqueName: \"kubernetes.io/projected/b1c53bd7-b699-4371-b434-d7882000083a-kube-api-access-2zr7m\") pod \"maas-api-revoke-sub-fa61a8da-tjpp6\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:52.961493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:52.960010    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:50:53.003728 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.003679    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:53.004049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.004015    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:53.037640 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.037578    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:50:53.062839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.062743    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:53.176909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.173571    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:53.307492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.304576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:50:53.307492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.305894    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:53.307492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.305948    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:53.307492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.306039    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:53.384502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.384369    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:53.384775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.384732    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:53.438120 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.438061    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:53.798645 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.798587    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:53.873853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.873793    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:53.874154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.874108    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:53.927811 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:53.927739    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:54.038674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.038624    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:54.131860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.054816    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:50:54.131860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.055328    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:50:54.175486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.174947    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:54.203558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.084423    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:54.203558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.203193    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:54.318492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.312763    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:54.321501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.320147    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:54.561559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.421113    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:54.561559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.421358    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:54.561559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.463585    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:54.715520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.715366    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:54.715520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.715429    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:54.753495 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:50:54.731607    4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.068s"
Oct 30 19:50:54.794778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.794732    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"]
Oct 30 19:50:54.819639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.819518    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:50:54.847993 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.847950    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:54.848300 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.848264    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:54.944643 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:54.944582    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:55.046262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.046218    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:50:55.101547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.100126    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:50:55.117497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.108121    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:50:55.129737 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.128025    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:55.142297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.142246    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:50:55.159555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.159152    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:50:55.184356 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.183945    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:55.243849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.243817    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:50:55.267626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.267592    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:50:55.274766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.274310    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:50:55.280449 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.280402    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:50:55.393899 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.393668    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:50:55.394242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.394206    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 19:50:55.613109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.613062    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:50:55.613329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.613130    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 19:50:55.749241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.749187    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6" event={"ID":"b1c53bd7-b699-4371-b434-d7882000083a","Type":"ContainerStarted","Data":"95a5c2859c9aba2a8cecfcbd28e416060dd8259048d4251800f73fca8569912d"}
Oct 30 19:50:55.749602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.749555    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6" event={"ID":"b1c53bd7-b699-4371-b434-d7882000083a","Type":"ContainerStarted","Data":"2b1666e43b74e0ea21d6bf4978d9559dcf4b17c1645ae57204811073b6a80059"}
Oct 30 19:50:55.790077 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.789964    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6" podStartSLOduration=4.789941812 podStartE2EDuration="4.789941812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:55.788971091 +0000 UTC m=+980.289674821" watchObservedRunningTime="2026-10-30 19:50:55.789941812 +0000 UTC m=+980.290645537"
Oct 30 19:50:55.846614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.846568    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:50:55.846841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:55.846649    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:50:56.502651 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.498438    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"]
Oct 30 19:50:56.526644 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.525871    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.604501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.604447    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"]
Oct 30 19:50:56.623671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.609516    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:56.623671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.613946    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:56.635963 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.634437    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.635963 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.634635    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.635963 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.634726    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46tdl\" (UniqueName: \"kubernetes.io/projected/4f99933f-b038-49a6-a443-166cd10e00ea-kube-api-access-46tdl\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.674537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.663322    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 19:50:56.709808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.709770    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:56.717487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.710351    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:56.717487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.713505    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:50:56.736193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.736155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.736551 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.736525    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.736743 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.736728    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46tdl\" (UniqueName: \"kubernetes.io/projected/4f99933f-b038-49a6-a443-166cd10e00ea-kube-api-access-46tdl\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.737335 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.737303    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.741178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.741147    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.760028 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.758738    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46tdl\" (UniqueName: \"kubernetes.io/projected/4f99933f-b038-49a6-a443-166cd10e00ea-kube-api-access-46tdl\") pod \"maas-api-revoke-sub-5e21050a-l7crz\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.765188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.765153    4519 generic.go:356] "Generic (PLEG): container finished" podID="b1c53bd7-b699-4371-b434-d7882000083a" containerID="95a5c2859c9aba2a8cecfcbd28e416060dd8259048d4251800f73fca8569912d" exitCode=0
Oct 30 19:50:56.766891 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.765746    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6" event={"ID":"b1c53bd7-b699-4371-b434-d7882000083a","Type":"ContainerDied","Data":"95a5c2859c9aba2a8cecfcbd28e416060dd8259048d4251800f73fca8569912d"}
Oct 30 19:50:56.999300 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.999261    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:50:56.999635 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:56.999302    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:57.000779 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.000681    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:57.001722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.001704    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 19:50:57.334888 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:57.334849    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f99933f_b038_49a6_a443_166cd10e00ea.slice/crio-04f24f0856fd6e3dbe75155e9ffabdc185decc362809e3eaf9e69a58c37fa993 WatchSource:0}: Error finding container 04f24f0856fd6e3dbe75155e9ffabdc185decc362809e3eaf9e69a58c37fa993: Status 404 returned error can't find the container with id 04f24f0856fd6e3dbe75155e9ffabdc185decc362809e3eaf9e69a58c37fa993
Oct 30 19:50:57.337049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.337020    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"]
Oct 30 19:50:57.520502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.517565    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"]
Oct 30 19:50:57.524852 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.524809    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.548996 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.548732    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"]
Oct 30 19:50:57.550450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.550418    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.550756 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.550732    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.556400 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.555272    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.556663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.556637    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.556900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.556884    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.557035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.557021    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwzh\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-kube-api-access-7nwzh\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.557165 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.557152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.557336 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.557322    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.557636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.557618    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.570628 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.569948    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"]
Oct 30 19:50:57.575072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.575035    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.598499 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.598441    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"]
Oct 30 19:50:57.659711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.659673    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.659900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.659762    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454c8916-e204-493e-9650-88be170d9224-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.659900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.659831    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpflf\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-kube-api-access-qpflf\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.660055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.659927    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.659955    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.660135 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660050    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.660135 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660118    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660145    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660258 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660236    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660261    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660302    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454c8916-e204-493e-9650-88be170d9224-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.660417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660341    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.660417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660403    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660441    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwzh\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-kube-api-access-7nwzh\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660506    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660595    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.660663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660622    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.660663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660648    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.660811 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.660787    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.661213 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.661178    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.661437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.661420    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.661670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.661651    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.662148 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.662123    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.664128 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.664102    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.664364 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.664340    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.681017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.680944    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.681177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.681080    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwzh\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-kube-api-access-7nwzh\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.761818 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.761767    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.761818 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.761818    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.761901    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.761959    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454c8916-e204-493e-9650-88be170d9224-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.761981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpflf\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-kube-api-access-qpflf\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762019    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762082    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762351 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762209    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454c8916-e204-493e-9650-88be170d9224-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762351 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762252    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762351 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762278    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762566 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762533    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762685 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762659    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762737    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454c8916-e204-493e-9650-88be170d9224-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.762813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.762795    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.764781 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.764757    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.765051 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.765029    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454c8916-e204-493e-9650-88be170d9224-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.776613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.776570    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz" event={"ID":"4f99933f-b038-49a6-a443-166cd10e00ea","Type":"ContainerStarted","Data":"1a900b66f938659340d4a0a00b282e0af198458c400f0d309c1f0d04abf7caea"}
Oct 30 19:50:57.776821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.776636    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz" event={"ID":"4f99933f-b038-49a6-a443-166cd10e00ea","Type":"ContainerStarted","Data":"04f24f0856fd6e3dbe75155e9ffabdc185decc362809e3eaf9e69a58c37fa993"}
Oct 30 19:50:57.783842 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.783810    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.785095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.785059    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpflf\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-kube-api-access-qpflf\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.820820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.820747    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz" podStartSLOduration=1.820726227 podStartE2EDuration="1.820726227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:57.815425804 +0000 UTC m=+982.316129529" watchObservedRunningTime="2026-10-30 19:50:57.820726227 +0000 UTC m=+982.321429952"
Oct 30 19:50:57.868242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.868154    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:57.900377 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.900336    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:57.953597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:57.953393    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:58.075711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.074967    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-xks-ca\") pod \"b1c53bd7-b699-4371-b434-d7882000083a\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") "
Oct 30 19:50:58.075711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.075093    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-openshift-ca\") pod \"b1c53bd7-b699-4371-b434-d7882000083a\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") "
Oct 30 19:50:58.075711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.075149    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c53bd7-b699-4371-b434-d7882000083a-kube-api-access-2zr7m\" (UniqueName: \"kubernetes.io/projected/b1c53bd7-b699-4371-b434-d7882000083a-kube-api-access-2zr7m\") pod \"b1c53bd7-b699-4371-b434-d7882000083a\" (UID: \"b1c53bd7-b699-4371-b434-d7882000083a\") "
Oct 30 19:50:58.076025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.075960    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-openshift-ca" pod "b1c53bd7-b699-4371-b434-d7882000083a" (UID: "b1c53bd7-b699-4371-b434-d7882000083a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:58.078006 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.077973    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-xks-ca" pod "b1c53bd7-b699-4371-b434-d7882000083a" (UID: "b1c53bd7-b699-4371-b434-d7882000083a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:50:58.078006 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.078000    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c53bd7-b699-4371-b434-d7882000083a-kube-api-access-2zr7m" pod "b1c53bd7-b699-4371-b434-d7882000083a" (UID: "b1c53bd7-b699-4371-b434-d7882000083a"). InnerVolumeSpecName "kube-api-access-2zr7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:58.177214 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.177167    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:50:58.177214 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.177214    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c53bd7-b699-4371-b434-d7882000083a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:50:58.177525 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.177231    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zr7m\" (UniqueName: \"kubernetes.io/projected/b1c53bd7-b699-4371-b434-d7882000083a-kube-api-access-2zr7m\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:50:58.216589 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:58.216556    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0fea40_0fee_47ae_bdef_739eeb5b8896.slice/crio-296038a19e243d7b08689f4bcce33b59a5e2dc16e79fcc30c23b5ff0fea5ea67 WatchSource:0}: Error finding container 296038a19e243d7b08689f4bcce33b59a5e2dc16e79fcc30c23b5ff0fea5ea67: Status 404 returned error can't find the container with id 296038a19e243d7b08689f4bcce33b59a5e2dc16e79fcc30c23b5ff0fea5ea67
Oct 30 19:50:58.218198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.218167    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"]
Oct 30 19:50:58.219654 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.219618    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:58.219777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.219703    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:58.219777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.219736    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:58.262491 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:50:58.262436    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454c8916_e204_493e_9650_88be170d9224.slice/crio-384652320e766e2b2f734858125977c8ffcee649998cc4363452082b7abc4c1e WatchSource:0}: Error finding container 384652320e766e2b2f734858125977c8ffcee649998cc4363452082b7abc4c1e: Status 404 returned error can't find the container with id 384652320e766e2b2f734858125977c8ffcee649998cc4363452082b7abc4c1e
Oct 30 19:50:58.265031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.264974    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"]
Oct 30 19:50:58.265995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.265953    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:58.266107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.266047    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:58.266107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.266094    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:58.792975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.792928    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" event={"ID":"1a0fea40-0fee-47ae-bdef-739eeb5b8896","Type":"ContainerStarted","Data":"b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428"}
Oct 30 19:50:58.793564 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.793006    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" event={"ID":"1a0fea40-0fee-47ae-bdef-739eeb5b8896","Type":"ContainerStarted","Data":"296038a19e243d7b08689f4bcce33b59a5e2dc16e79fcc30c23b5ff0fea5ea67"}
Oct 30 19:50:58.795588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.795550    4519 generic.go:356] "Generic (PLEG): container finished" podID="4f99933f-b038-49a6-a443-166cd10e00ea" containerID="1a900b66f938659340d4a0a00b282e0af198458c400f0d309c1f0d04abf7caea" exitCode=0
Oct 30 19:50:58.795737 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.795638    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz" event={"ID":"4f99933f-b038-49a6-a443-166cd10e00ea","Type":"ContainerDied","Data":"1a900b66f938659340d4a0a00b282e0af198458c400f0d309c1f0d04abf7caea"}
Oct 30 19:50:58.797672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.797648    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"
Oct 30 19:50:58.797939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.797655    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6" event={"ID":"b1c53bd7-b699-4371-b434-d7882000083a","Type":"ContainerDied","Data":"2b1666e43b74e0ea21d6bf4978d9559dcf4b17c1645ae57204811073b6a80059"}
Oct 30 19:50:58.798063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.797970    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b1666e43b74e0ea21d6bf4978d9559dcf4b17c1645ae57204811073b6a80059"
Oct 30 19:50:58.799937 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.799895    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" event={"ID":"454c8916-e204-493e-9650-88be170d9224","Type":"ContainerStarted","Data":"bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c"}
Oct 30 19:50:58.800123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.799976    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" event={"ID":"454c8916-e204-493e-9650-88be170d9224","Type":"ContainerStarted","Data":"384652320e766e2b2f734858125977c8ffcee649998cc4363452082b7abc4c1e"}
Oct 30 19:50:58.843710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.843639    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podStartSLOduration=1.843624194 podStartE2EDuration="1.843624194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:58.840682284 +0000 UTC m=+983.341386009" watchObservedRunningTime="2026-10-30 19:50:58.843624194 +0000 UTC m=+983.344327918"
Oct 30 19:50:58.868686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.868632    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:50:58.901092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.901046    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:50:58.914447 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:58.914370    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podStartSLOduration=1.914345403 podStartE2EDuration="1.914345403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:58.911866528 +0000 UTC m=+983.412570253" watchObservedRunningTime="2026-10-30 19:50:58.914345403 +0000 UTC m=+983.415049130"
Oct 30 19:50:59.877071 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:59.877031    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:50:59.877963 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:59.877667    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 19:50:59.920124 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:59.919616    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:50:59.920124 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:50:59.919688    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:51:00.225964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.225931    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:51:00.237112 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.237074    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-xks-ca\") pod \"4f99933f-b038-49a6-a443-166cd10e00ea\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") "
Oct 30 19:51:00.237323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.237266    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-openshift-ca\") pod \"4f99933f-b038-49a6-a443-166cd10e00ea\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") "
Oct 30 19:51:00.237323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.237320    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f99933f-b038-49a6-a443-166cd10e00ea-kube-api-access-46tdl\" (UniqueName: \"kubernetes.io/projected/4f99933f-b038-49a6-a443-166cd10e00ea-kube-api-access-46tdl\") pod \"4f99933f-b038-49a6-a443-166cd10e00ea\" (UID: \"4f99933f-b038-49a6-a443-166cd10e00ea\") "
Oct 30 19:51:00.237736 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.237704    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-openshift-ca" pod "4f99933f-b038-49a6-a443-166cd10e00ea" (UID: "4f99933f-b038-49a6-a443-166cd10e00ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:00.238188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.237980    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:00.239705 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.239637    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-xks-ca" pod "4f99933f-b038-49a6-a443-166cd10e00ea" (UID: "4f99933f-b038-49a6-a443-166cd10e00ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:00.239827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.239717    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f99933f-b038-49a6-a443-166cd10e00ea-kube-api-access-46tdl" pod "4f99933f-b038-49a6-a443-166cd10e00ea" (UID: "4f99933f-b038-49a6-a443-166cd10e00ea"). InnerVolumeSpecName "kube-api-access-46tdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:00.339762 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.339709    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f99933f-b038-49a6-a443-166cd10e00ea-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:00.339762 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.339748    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46tdl\" (UniqueName: \"kubernetes.io/projected/4f99933f-b038-49a6-a443-166cd10e00ea-kube-api-access-46tdl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:00.828078 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.828029    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz" event={"ID":"4f99933f-b038-49a6-a443-166cd10e00ea","Type":"ContainerDied","Data":"04f24f0856fd6e3dbe75155e9ffabdc185decc362809e3eaf9e69a58c37fa993"}
Oct 30 19:51:00.828340 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.828324    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04f24f0856fd6e3dbe75155e9ffabdc185decc362809e3eaf9e69a58c37fa993"
Oct 30 19:51:00.828422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.828164    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"
Oct 30 19:51:00.904495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.903934    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:51:00.904495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.904003    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:51:00.913529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.910609    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:51:00.913529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:00.910683    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 19:51:01.875490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:01.874892    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:51:01.875490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:01.874962    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 19:51:01.908492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:01.907602    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:51:01.908492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:01.907679    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:51:02.237887 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:51:02.237847    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:51:02.872489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:02.871531    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:51:02.872489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:02.871620    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 19:51:02.903860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:02.903822    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:51:02.904124 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:02.904102    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:51:03.875098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:03.874623    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:51:03.875098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:03.874690    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 19:51:03.901839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:03.901799    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:51:03.902088 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:03.902062    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:51:04.869221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:04.869156    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:51:04.869532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:04.869485    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 19:51:04.901097 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:04.901055    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:51:04.901634 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:04.901131    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:51:05.878417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:05.876518    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:51:05.878417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:05.876973    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:51:05.878417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:05.878105    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:51:06.295106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:06.295052    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:51:06.296716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:06.296687    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:51:06.298794 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:06.298767    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 19:51:12.841589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.841540    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"]
Oct 30 19:51:12.842192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.842135    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4f99933f-b038-49a6-a443-166cd10e00ea" containerName="revoke-keys"
Oct 30 19:51:12.842192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.842147    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f99933f-b038-49a6-a443-166cd10e00ea" containerName="revoke-keys"
Oct 30 19:51:12.842192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.842177    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b1c53bd7-b699-4371-b434-d7882000083a" containerName="revoke-keys"
Oct 30 19:51:12.842192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.842182    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c53bd7-b699-4371-b434-d7882000083a" containerName="revoke-keys"
Oct 30 19:51:12.842418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.842274    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f99933f-b038-49a6-a443-166cd10e00ea" containerName="revoke-keys"
Oct 30 19:51:12.842418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.842287    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c53bd7-b699-4371-b434-d7882000083a" containerName="revoke-keys"
Oct 30 19:51:12.846035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.846009    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850460    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850551    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q825f\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-kube-api-access-q825f\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850666    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850725    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850755    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850779    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850829    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850917 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850898    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.850994 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.850978    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.890691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.890653    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"]
Oct 30 19:51:12.947380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.947332    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"]
Oct 30 19:51:12.952134 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952092    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952150    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952194    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952267    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952314    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952382    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952586 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952448    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q825f\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-kube-api-access-q825f\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952592    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.952696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.952668    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.953099 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.953074    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.953425 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.953402    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.953839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.953804    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.954574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.954548    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.954574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.954564    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.956269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.956237    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:12.957663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.957587    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.959128 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.959096    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:12.991775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:12.991740    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"]
Oct 30 19:51:13.000439 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.000405    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:13.002461 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.002429    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q825f\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-kube-api-access-q825f\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:13.054158 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054111    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054167    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054259    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054347 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054289    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054428    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054548    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq745\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-kube-api-access-mq745\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054637 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054621    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e971a-1282-4810-a3e4-afe14b00225a-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054739    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e971a-1282-4810-a3e4-afe14b00225a-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.054801 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.054787    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.122838 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.122735    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"]
Oct 30 19:51:13.127617 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.127594    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.155802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155399    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"]
Oct 30 19:51:13.155802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155648    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.155802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155687    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.155802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155715    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.155802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155741    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.155802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155825    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d5626f-14c3-45e7-8e08-e732f45b5316-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155898    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155927    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.155976    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156002    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq745\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-kube-api-access-mq745\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156031    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfr5\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-kube-api-access-swfr5\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156086    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e971a-1282-4810-a3e4-afe14b00225a-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156110    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.156281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156192    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e971a-1282-4810-a3e4-afe14b00225a-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.156760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156585    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.156874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156852    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.156930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.156907    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.157123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.157098    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.157329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.157309    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.157524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.157120    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.157697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.157667    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.157809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.157711    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e971a-1282-4810-a3e4-afe14b00225a-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.157809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.157729    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.159242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.159190    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.161094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.161044    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e971a-1282-4810-a3e4-afe14b00225a-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.163091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.162869    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:13.190826 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.190797    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq745\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-kube-api-access-mq745\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.191138 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.191120    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.260009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.259963    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260247 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260047    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d5626f-14c3-45e7-8e08-e732f45b5316-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260247 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260182    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260242    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260283    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfr5\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-kube-api-access-swfr5\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260339    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260452    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260691    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.260926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.260741    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.261985 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.261950    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.262983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.262955    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d5626f-14c3-45e7-8e08-e732f45b5316-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.263224 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.263201    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.263783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.263759    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.264016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.263997    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.266162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.266033    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.267438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.267403    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.284162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.284123    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfr5\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-kube-api-access-swfr5\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.284506 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.284426    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.332781 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.332743    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:13.443024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.442986    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:13.505217 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:13.504174    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808e5138_e9a8_4ccf_97e0_a9ddd57bfb40.slice/crio-e735576afbf2789f1c658e19f012e6fcdcc79e84f70e4eb6dbb5c2609af7f9aa WatchSource:0}: Error finding container e735576afbf2789f1c658e19f012e6fcdcc79e84f70e4eb6dbb5c2609af7f9aa: Status 404 returned error can't find the container with id e735576afbf2789f1c658e19f012e6fcdcc79e84f70e4eb6dbb5c2609af7f9aa
Oct 30 19:51:13.507117 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.507087    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"]
Oct 30 19:51:13.508006 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.507962    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.508125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.508056    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.508182 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.508130    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.660769 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:13.660727    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5e971a_1282_4810_a3e4_afe14b00225a.slice/crio-dafdafddd92d6b1e556495852d99e4496aea7af7d2aba0358532a686d19cdfd3 WatchSource:0}: Error finding container dafdafddd92d6b1e556495852d99e4496aea7af7d2aba0358532a686d19cdfd3: Status 404 returned error can't find the container with id dafdafddd92d6b1e556495852d99e4496aea7af7d2aba0358532a686d19cdfd3
Oct 30 19:51:13.665357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.664817    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.665357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.664904    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.665357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.664952    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.667969 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.667939    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"]
Oct 30 19:51:13.791813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.785735    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"]
Oct 30 19:51:13.801383 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.801348    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.801520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.801427    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:13.801520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:13.801459    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:14.059686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.059598    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" event={"ID":"af5e971a-1282-4810-a3e4-afe14b00225a","Type":"ContainerStarted","Data":"4f569f56280fa93a7f29a38ae77e644402bc6acbe6691e5cebc02c0172e3291e"}
Oct 30 19:51:14.059686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.059682    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" event={"ID":"af5e971a-1282-4810-a3e4-afe14b00225a","Type":"ContainerStarted","Data":"dafdafddd92d6b1e556495852d99e4496aea7af7d2aba0358532a686d19cdfd3"}
Oct 30 19:51:14.083332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.082332    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" event={"ID":"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40","Type":"ContainerStarted","Data":"156a1b4bf3fa72f380673944da2139ec03bc4002efcc19caab93003e8fe14043"}
Oct 30 19:51:14.083332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.082408    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" event={"ID":"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40","Type":"ContainerStarted","Data":"e735576afbf2789f1c658e19f012e6fcdcc79e84f70e4eb6dbb5c2609af7f9aa"}
Oct 30 19:51:14.087218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.086350    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" event={"ID":"79d5626f-14c3-45e7-8e08-e732f45b5316","Type":"ContainerStarted","Data":"ae6132e449f1c4200ebd6a277e76cef1f7fb1416baeae845d48efc8adf78dbfc"}
Oct 30 19:51:14.087218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.086414    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" event={"ID":"79d5626f-14c3-45e7-8e08-e732f45b5316","Type":"ContainerStarted","Data":"372565fc46ff2a87fc444d2de20d2e46aafecf018094d902334eb604819b623e"}
Oct 30 19:51:14.110479 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.110394    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podStartSLOduration=2.110378983 podStartE2EDuration="2.110378983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:14.10830161 +0000 UTC m=+998.609005338" watchObservedRunningTime="2026-10-30 19:51:14.110378983 +0000 UTC m=+998.611082709"
Oct 30 19:51:14.155486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.155403    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podStartSLOduration=1.155378848 podStartE2EDuration="1.155378848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:14.150228257 +0000 UTC m=+998.650931983" watchObservedRunningTime="2026-10-30 19:51:14.155378848 +0000 UTC m=+998.656082574"
Oct 30 19:51:14.168285 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.168235    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:14.169659 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.169628    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:14.169818 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.169679    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:14.197646 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.197568    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podStartSLOduration=2.197544904 podStartE2EDuration="2.197544904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:14.196384264 +0000 UTC m=+998.697087989" watchObservedRunningTime="2026-10-30 19:51:14.197544904 +0000 UTC m=+998.698248631"
Oct 30 19:51:14.333658 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.333571    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:14.334895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.334865    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:14.335000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.334931    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:14.443195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.443157    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:14.444865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.444837    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:14.444999 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:14.444890    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:15.167414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:15.166922    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:15.167414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:15.166991    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:15.343884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:15.343818    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:15.344123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:15.343909    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:15.443844 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:15.443741    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:15.443844 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:15.443822    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:16.180491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:16.179623    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:16.180491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:16.179720    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:16.333967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:16.333921    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:16.334320 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:16.334285    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:16.446072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:16.445964    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:16.446331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:16.446298    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:17.197849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:17.197812    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:17.212660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:17.212606    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:17.344002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:17.343961    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:17.353607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:17.353554    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:17.448226 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:17.448129    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:17.448546 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:17.448516    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:17.966495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:17.964575    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:18.184529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:18.167943    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:18.184529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:18.168024    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:18.340557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:18.339489    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:18.340557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:18.339559    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:18.443490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:18.443324    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:18.443490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:18.443387    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:19.183492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:19.182717    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:19.183492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:19.182799    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:19.336492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:19.335929    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:19.336492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:19.336017    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:19.445059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:19.444954    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:19.445677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:19.445640    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:20.170492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:20.170244    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:20.170492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:20.170339    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:20.336898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:20.336856    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:20.337199 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:20.337168    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:20.401307 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:20.401180    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:20.460789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:20.444185    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:20.460789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:20.444259    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:21.170493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:21.169954    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:51:21.265807 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:21.244538    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 19:51:21.334486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:21.334427    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:21.334800 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:21.334769    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:21.445623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:21.445512    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:21.445623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:21.445584    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:22.396276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:22.396226    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:51:22.398638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:22.398578    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:51:22.446056 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:22.446012    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:51:22.446369 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:22.446336    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:51:23.034525 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.034453    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:23.228787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.228742    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:23.229091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.229051    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:23.592917 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.592820    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:23.595552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.594188    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:23.596997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.596957    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:51:23.925605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.925141    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:23.926202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.926177    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:23.927293 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.927262    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 19:51:23.937809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.937768    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:23.939461 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.939437    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:23.950511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:23.948632    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:51:24.506015 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.505978    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"]
Oct 30 19:51:24.510487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.510438    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.530537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.530508    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"]
Oct 30 19:51:24.536081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.536049    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:24.536352 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.536333    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:51:24.591719 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.591673    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz67n\" (UniqueName: \"kubernetes.io/projected/1c3472ce-3ec5-4c49-adca-cea04950ebb1-kube-api-access-wz67n\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.591929 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.591908    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.592155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.592132    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.693695 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.693649    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz67n\" (UniqueName: \"kubernetes.io/projected/1c3472ce-3ec5-4c49-adca-cea04950ebb1-kube-api-access-wz67n\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.694173 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.693845    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.694173 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.693990    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.694742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.694712    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.696627 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.696597    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.713511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.713459    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz67n\" (UniqueName: \"kubernetes.io/projected/1c3472ce-3ec5-4c49-adca-cea04950ebb1-kube-api-access-wz67n\") pod \"maas-api-revoke-sub-1095500b-qtjr5\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:24.825425 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:24.825331    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:25.142715 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:25.142680    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3472ce_3ec5_4c49_adca_cea04950ebb1.slice/crio-12e908f71725210e37dd9a1116c6e5d7b3c05aaa15135d551d0d545c4cf7fd38 WatchSource:0}: Error finding container 12e908f71725210e37dd9a1116c6e5d7b3c05aaa15135d551d0d545c4cf7fd38: Status 404 returned error can't find the container with id 12e908f71725210e37dd9a1116c6e5d7b3c05aaa15135d551d0d545c4cf7fd38
Oct 30 19:51:25.144741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:25.144700    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"]
Oct 30 19:51:25.495048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:25.495004    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5" event={"ID":"1c3472ce-3ec5-4c49-adca-cea04950ebb1","Type":"ContainerStarted","Data":"98b75d6163f05b2720f52496cdec17be54914e454bc19794ff67c6035b9f709b"}
Oct 30 19:51:25.495256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:25.495077    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5" event={"ID":"1c3472ce-3ec5-4c49-adca-cea04950ebb1","Type":"ContainerStarted","Data":"12e908f71725210e37dd9a1116c6e5d7b3c05aaa15135d551d0d545c4cf7fd38"}
Oct 30 19:51:25.539324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:25.539265    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5" podStartSLOduration=1.539247217 podStartE2EDuration="1.539247217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:25.533655779 +0000 UTC m=+1010.034359503" watchObservedRunningTime="2026-10-30 19:51:25.539247217 +0000 UTC m=+1010.039950940"
Oct 30 19:51:26.025861 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.025816    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"]
Oct 30 19:51:26.030982 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.030959    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.056171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.056128    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"]
Oct 30 19:51:26.117854 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.117802    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.118172 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.118131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72kl\" (UniqueName: \"kubernetes.io/projected/192871e4-b312-4780-95b5-20aa6ef86770-kube-api-access-v72kl\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.118264 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.118199    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.219936 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.219898    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72kl\" (UniqueName: \"kubernetes.io/projected/192871e4-b312-4780-95b5-20aa6ef86770-kube-api-access-v72kl\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.219936 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.219945    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.220218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.220060    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.220787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.220757    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.222620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.222571    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.240020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.239988    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72kl\" (UniqueName: \"kubernetes.io/projected/192871e4-b312-4780-95b5-20aa6ef86770-kube-api-access-v72kl\") pod \"maas-api-revoke-sub-425381a4-s6z98\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.346413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.346378    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:26.508982 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.508894    4519 generic.go:356] "Generic (PLEG): container finished" podID="1c3472ce-3ec5-4c49-adca-cea04950ebb1" containerID="98b75d6163f05b2720f52496cdec17be54914e454bc19794ff67c6035b9f709b" exitCode=0
Oct 30 19:51:26.509147 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.509027    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5" event={"ID":"1c3472ce-3ec5-4c49-adca-cea04950ebb1","Type":"ContainerDied","Data":"98b75d6163f05b2720f52496cdec17be54914e454bc19794ff67c6035b9f709b"}
Oct 30 19:51:26.678129 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:26.678076    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192871e4_b312_4780_95b5_20aa6ef86770.slice/crio-aa7460671bb445ef44e95b7b27d3ea7340332a645d6c1427b5db804e09546d2b WatchSource:0}: Error finding container aa7460671bb445ef44e95b7b27d3ea7340332a645d6c1427b5db804e09546d2b: Status 404 returned error can't find the container with id aa7460671bb445ef44e95b7b27d3ea7340332a645d6c1427b5db804e09546d2b
Oct 30 19:51:26.681310 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:26.681170    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"]
Oct 30 19:51:27.519443 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.519404    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98" event={"ID":"192871e4-b312-4780-95b5-20aa6ef86770","Type":"ContainerStarted","Data":"20119578225c321e2db83808ea324e9ff0636a12a467501d1f030b8c9d714591"}
Oct 30 19:51:27.519963 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.519481    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98" event={"ID":"192871e4-b312-4780-95b5-20aa6ef86770","Type":"ContainerStarted","Data":"aa7460671bb445ef44e95b7b27d3ea7340332a645d6c1427b5db804e09546d2b"}
Oct 30 19:51:27.560579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.560517    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98" podStartSLOduration=2.560500675 podStartE2EDuration="2.560500675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:27.558566261 +0000 UTC m=+1012.059269984" watchObservedRunningTime="2026-10-30 19:51:27.560500675 +0000 UTC m=+1012.061204395"
Oct 30 19:51:27.709179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.709149    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:27.845798 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.845698    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3472ce-3ec5-4c49-adca-cea04950ebb1-kube-api-access-wz67n\" (UniqueName: \"kubernetes.io/projected/1c3472ce-3ec5-4c49-adca-cea04950ebb1-kube-api-access-wz67n\") pod \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") "
Oct 30 19:51:27.845987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.845947    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-openshift-ca\") pod \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") "
Oct 30 19:51:27.846102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.846086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-xks-ca\") pod \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\" (UID: \"1c3472ce-3ec5-4c49-adca-cea04950ebb1\") "
Oct 30 19:51:27.846295 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.846264    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-openshift-ca" pod "1c3472ce-3ec5-4c49-adca-cea04950ebb1" (UID: "1c3472ce-3ec5-4c49-adca-cea04950ebb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:27.846765 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.846738    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:27.847949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.847917    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3472ce-3ec5-4c49-adca-cea04950ebb1-kube-api-access-wz67n" pod "1c3472ce-3ec5-4c49-adca-cea04950ebb1" (UID: "1c3472ce-3ec5-4c49-adca-cea04950ebb1"). InnerVolumeSpecName "kube-api-access-wz67n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:27.848221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.848200    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-xks-ca" pod "1c3472ce-3ec5-4c49-adca-cea04950ebb1" (UID: "1c3472ce-3ec5-4c49-adca-cea04950ebb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:27.948234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.948183    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz67n\" (UniqueName: \"kubernetes.io/projected/1c3472ce-3ec5-4c49-adca-cea04950ebb1-kube-api-access-wz67n\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:27.948234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:27.948230    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3472ce-3ec5-4c49-adca-cea04950ebb1-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:28.533724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:28.533681    4519 generic.go:356] "Generic (PLEG): container finished" podID="192871e4-b312-4780-95b5-20aa6ef86770" containerID="20119578225c321e2db83808ea324e9ff0636a12a467501d1f030b8c9d714591" exitCode=0
Oct 30 19:51:28.534224 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:28.533764    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98" event={"ID":"192871e4-b312-4780-95b5-20aa6ef86770","Type":"ContainerDied","Data":"20119578225c321e2db83808ea324e9ff0636a12a467501d1f030b8c9d714591"}
Oct 30 19:51:28.535753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:28.535720    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"
Oct 30 19:51:28.535897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:28.535756    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5" event={"ID":"1c3472ce-3ec5-4c49-adca-cea04950ebb1","Type":"ContainerDied","Data":"12e908f71725210e37dd9a1116c6e5d7b3c05aaa15135d551d0d545c4cf7fd38"}
Oct 30 19:51:28.535897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:28.535818    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e908f71725210e37dd9a1116c6e5d7b3c05aaa15135d551d0d545c4cf7fd38"
Oct 30 19:51:29.721334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.721300    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:29.817045 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.816937    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192871e4-b312-4780-95b5-20aa6ef86770-kube-api-access-v72kl\" (UniqueName: \"kubernetes.io/projected/192871e4-b312-4780-95b5-20aa6ef86770-kube-api-access-v72kl\") pod \"192871e4-b312-4780-95b5-20aa6ef86770\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") "
Oct 30 19:51:29.817235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.817086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-openshift-ca\") pod \"192871e4-b312-4780-95b5-20aa6ef86770\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") "
Oct 30 19:51:29.817235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.817190    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-xks-ca\") pod \"192871e4-b312-4780-95b5-20aa6ef86770\" (UID: \"192871e4-b312-4780-95b5-20aa6ef86770\") "
Oct 30 19:51:29.818811 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.818771    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-openshift-ca" pod "192871e4-b312-4780-95b5-20aa6ef86770" (UID: "192871e4-b312-4780-95b5-20aa6ef86770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:29.820822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.820786    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-xks-ca" pod "192871e4-b312-4780-95b5-20aa6ef86770" (UID: "192871e4-b312-4780-95b5-20aa6ef86770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:29.827037 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.826996    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192871e4-b312-4780-95b5-20aa6ef86770-kube-api-access-v72kl" pod "192871e4-b312-4780-95b5-20aa6ef86770" (UID: "192871e4-b312-4780-95b5-20aa6ef86770"). InnerVolumeSpecName "kube-api-access-v72kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:29.919105 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.919061    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:29.919105 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.919101    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192871e4-b312-4780-95b5-20aa6ef86770-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:29.919105 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:29.919112    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72kl\" (UniqueName: \"kubernetes.io/projected/192871e4-b312-4780-95b5-20aa6ef86770-kube-api-access-v72kl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:30.554874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:30.554841    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"
Oct 30 19:51:30.555055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:30.554840    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98" event={"ID":"192871e4-b312-4780-95b5-20aa6ef86770","Type":"ContainerDied","Data":"aa7460671bb445ef44e95b7b27d3ea7340332a645d6c1427b5db804e09546d2b"}
Oct 30 19:51:30.555055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:30.555009    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa7460671bb445ef44e95b7b27d3ea7340332a645d6c1427b5db804e09546d2b"
Oct 30 19:51:32.222159 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:51:32.222120    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:51:33.113014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.112978    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"]
Oct 30 19:51:33.113652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.113634    4519 cpu_manager.go:414] "containerMap: removing container" podUID="192871e4-b312-4780-95b5-20aa6ef86770" containerName="revoke-keys"
Oct 30 19:51:33.113710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.113653    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192871e4-b312-4780-95b5-20aa6ef86770" containerName="revoke-keys"
Oct 30 19:51:33.113710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.113667    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1c3472ce-3ec5-4c49-adca-cea04950ebb1" containerName="revoke-keys"
Oct 30 19:51:33.113710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.113672    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3472ce-3ec5-4c49-adca-cea04950ebb1" containerName="revoke-keys"
Oct 30 19:51:33.113856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.113781    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="192871e4-b312-4780-95b5-20aa6ef86770" containerName="revoke-keys"
Oct 30 19:51:33.113856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.113794    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3472ce-3ec5-4c49-adca-cea04950ebb1" containerName="revoke-keys"
Oct 30 19:51:33.118952 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.118922    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.145686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.145644    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"]
Oct 30 19:51:33.149891 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.149859    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:33.150139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.149969    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:51:33.176221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.176187    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.176423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.176292    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsgs5\" (UniqueName: \"kubernetes.io/projected/a48745e1-df7e-4d2e-8c16-29a22f25d533-kube-api-access-vsgs5\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.176566 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.176539    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.279837 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.279696    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgs5\" (UniqueName: \"kubernetes.io/projected/a48745e1-df7e-4d2e-8c16-29a22f25d533-kube-api-access-vsgs5\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.280332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.279937    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.280332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.280098    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.282055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.282001    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.292332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.283447    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.306217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.306166    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsgs5\" (UniqueName: \"kubernetes.io/projected/a48745e1-df7e-4d2e-8c16-29a22f25d533-kube-api-access-vsgs5\") pod \"maas-api-revoke-sub-52db9798-bhtf9\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.434328 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.434287    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:33.753483 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:33.753429    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48745e1_df7e_4d2e_8c16_29a22f25d533.slice/crio-698fc42ccc9c950b44ad78e9fc322eefae42a8d142a03e2a90df1f824136e57f WatchSource:0}: Error finding container 698fc42ccc9c950b44ad78e9fc322eefae42a8d142a03e2a90df1f824136e57f: Status 404 returned error can't find the container with id 698fc42ccc9c950b44ad78e9fc322eefae42a8d142a03e2a90df1f824136e57f
Oct 30 19:51:33.756540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:33.756514    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"]
Oct 30 19:51:34.596476 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:34.596403    4519 generic.go:356] "Generic (PLEG): container finished" podID="a48745e1-df7e-4d2e-8c16-29a22f25d533" containerID="3fb2cf00bcaa7f6ef37eed71e9b03278baf87eda15fdc1e75862ac28081d309a" exitCode=0
Oct 30 19:51:34.596932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:34.596530    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9" event={"ID":"a48745e1-df7e-4d2e-8c16-29a22f25d533","Type":"ContainerDied","Data":"3fb2cf00bcaa7f6ef37eed71e9b03278baf87eda15fdc1e75862ac28081d309a"}
Oct 30 19:51:34.596932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:34.596584    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9" event={"ID":"a48745e1-df7e-4d2e-8c16-29a22f25d533","Type":"ContainerStarted","Data":"698fc42ccc9c950b44ad78e9fc322eefae42a8d142a03e2a90df1f824136e57f"}
Oct 30 19:51:35.816926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.816898    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:35.925751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.925712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-xks-ca\") pod \"a48745e1-df7e-4d2e-8c16-29a22f25d533\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") "
Oct 30 19:51:35.925751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.925769    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-openshift-ca\") pod \"a48745e1-df7e-4d2e-8c16-29a22f25d533\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") "
Oct 30 19:51:35.926044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.925834    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48745e1-df7e-4d2e-8c16-29a22f25d533-kube-api-access-vsgs5\" (UniqueName: \"kubernetes.io/projected/a48745e1-df7e-4d2e-8c16-29a22f25d533-kube-api-access-vsgs5\") pod \"a48745e1-df7e-4d2e-8c16-29a22f25d533\" (UID: \"a48745e1-df7e-4d2e-8c16-29a22f25d533\") "
Oct 30 19:51:35.926219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.926191    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-openshift-ca" pod "a48745e1-df7e-4d2e-8c16-29a22f25d533" (UID: "a48745e1-df7e-4d2e-8c16-29a22f25d533"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:35.926489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.926454    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:35.928602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.928569    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-xks-ca" pod "a48745e1-df7e-4d2e-8c16-29a22f25d533" (UID: "a48745e1-df7e-4d2e-8c16-29a22f25d533"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:35.928751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:35.928645    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48745e1-df7e-4d2e-8c16-29a22f25d533-kube-api-access-vsgs5" pod "a48745e1-df7e-4d2e-8c16-29a22f25d533" (UID: "a48745e1-df7e-4d2e-8c16-29a22f25d533"). InnerVolumeSpecName "kube-api-access-vsgs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:36.028199 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:36.028147    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48745e1-df7e-4d2e-8c16-29a22f25d533-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:36.028199 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:36.028192    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsgs5\" (UniqueName: \"kubernetes.io/projected/a48745e1-df7e-4d2e-8c16-29a22f25d533-kube-api-access-vsgs5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:36.624372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:36.624326    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9" event={"ID":"a48745e1-df7e-4d2e-8c16-29a22f25d533","Type":"ContainerDied","Data":"698fc42ccc9c950b44ad78e9fc322eefae42a8d142a03e2a90df1f824136e57f"}
Oct 30 19:51:36.624372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:36.624349    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"
Oct 30 19:51:36.624625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:36.624386    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="698fc42ccc9c950b44ad78e9fc322eefae42a8d142a03e2a90df1f824136e57f"
Oct 30 19:51:38.998016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:38.997974    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"]
Oct 30 19:51:38.998667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:38.998650    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a48745e1-df7e-4d2e-8c16-29a22f25d533" containerName="revoke-keys"
Oct 30 19:51:38.998723 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:38.998668    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48745e1-df7e-4d2e-8c16-29a22f25d533" containerName="revoke-keys"
Oct 30 19:51:38.998804 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:38.998795    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48745e1-df7e-4d2e-8c16-29a22f25d533" containerName="revoke-keys"
Oct 30 19:51:39.002975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.002954    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.018239 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.018201    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:39.038817 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.038778    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:51:39.045597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.045554    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"]
Oct 30 19:51:39.075829 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.075783    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7527g\" (UniqueName: \"kubernetes.io/projected/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-kube-api-access-7527g\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.076091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.076048    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.076228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.076204    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.177767 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.177730    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7527g\" (UniqueName: \"kubernetes.io/projected/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-kube-api-access-7527g\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.177989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.177898    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.178047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.177996    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.178911 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.178876    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.180696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.180663    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.200445 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.200406    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7527g\" (UniqueName: \"kubernetes.io/projected/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-kube-api-access-7527g\") pod \"maas-api-revoke-sub-c0fa03f8-pf6nz\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.318864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.318772    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:39.633368 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:39.633320    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518e2010_cfb2_4726_a42f_4f0cfb7ff3a9.slice/crio-445fff388c06e09f6779c000fc69300126e838b674618cef3a90b2cf89868e49 WatchSource:0}: Error finding container 445fff388c06e09f6779c000fc69300126e838b674618cef3a90b2cf89868e49: Status 404 returned error can't find the container with id 445fff388c06e09f6779c000fc69300126e838b674618cef3a90b2cf89868e49
Oct 30 19:51:39.635783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.635755    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"]
Oct 30 19:51:39.651887 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.651847    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz" event={"ID":"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9","Type":"ContainerStarted","Data":"445fff388c06e09f6779c000fc69300126e838b674618cef3a90b2cf89868e49"}
Oct 30 19:51:39.764570 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.764523    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"]
Oct 30 19:51:39.773266 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.773226    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.798419 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.798380    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"]
Oct 30 19:51:39.892821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.892680    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjxk\" (UniqueName: \"kubernetes.io/projected/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-kube-api-access-mbjxk\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.893009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.892854    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.893074 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.893008    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.996042 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.995994    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjxk\" (UniqueName: \"kubernetes.io/projected/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-kube-api-access-mbjxk\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.996245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.996073    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.996329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.996244    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.996943 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.996914    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:39.999155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:39.999122    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:40.016323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.016289    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjxk\" (UniqueName: \"kubernetes.io/projected/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-kube-api-access-mbjxk\") pod \"maas-api-revoke-sub-1b467df3-trq2r\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:40.109923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.109880    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:40.391760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.391718    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"]
Oct 30 19:51:40.398738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.398709    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.427949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.427832    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"]
Oct 30 19:51:40.461363 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:40.461274    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27956871_d80b_4a5b_bfd5_3bbd0d5b6968.slice/crio-46f1ca7e3097037cfa695d2adc95c2bdcce19b4cdf3a8a42dab96346d2972b3b WatchSource:0}: Error finding container 46f1ca7e3097037cfa695d2adc95c2bdcce19b4cdf3a8a42dab96346d2972b3b: Status 404 returned error can't find the container with id 46f1ca7e3097037cfa695d2adc95c2bdcce19b4cdf3a8a42dab96346d2972b3b
Oct 30 19:51:40.467416 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.467384    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"]
Oct 30 19:51:40.505649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.505615    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.505816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.505703    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mp5b\" (UniqueName: \"kubernetes.io/projected/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-kube-api-access-2mp5b\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.505871 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.505846    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.607997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.607952    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.608222 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.608087    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mp5b\" (UniqueName: \"kubernetes.io/projected/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-kube-api-access-2mp5b\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.608222 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.608215    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.608858 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.608833    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.610876 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.610846    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.629026 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.628994    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mp5b\" (UniqueName: \"kubernetes.io/projected/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-kube-api-access-2mp5b\") pod \"maas-api-revoke-sub-3adc4e00-48wx9\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:40.665353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.665311    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r" event={"ID":"27956871-d80b-4a5b-bfd5-3bbd0d5b6968","Type":"ContainerStarted","Data":"46f1ca7e3097037cfa695d2adc95c2bdcce19b4cdf3a8a42dab96346d2972b3b"}
Oct 30 19:51:40.667534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.667450    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz" event={"ID":"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9","Type":"ContainerStarted","Data":"88363467deb7b23ce6a67db8c3ebf1604a2063e3a3f2a2870846764be752068c"}
Oct 30 19:51:40.715187 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:40.715091    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:41.028980 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:41.028941    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f383ca_df7f_4185_8aac_d6f3b12dd2c2.slice/crio-eaa83705d78b582e541b94ba09c085e4402fa25eacb4f4173b249662eb2ec621 WatchSource:0}: Error finding container eaa83705d78b582e541b94ba09c085e4402fa25eacb4f4173b249662eb2ec621: Status 404 returned error can't find the container with id eaa83705d78b582e541b94ba09c085e4402fa25eacb4f4173b249662eb2ec621
Oct 30 19:51:41.031427 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.031398    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"]
Oct 30 19:51:41.107725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.107686    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"]
Oct 30 19:51:41.114021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.113973    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.132564 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.132528    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"]
Oct 30 19:51:41.217270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.217224    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.217438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.217347    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.217604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.217565    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wmz\" (UniqueName: \"kubernetes.io/projected/05df8544-fb87-4486-baa5-e2d107f355ee-kube-api-access-k5wmz\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.319432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.319336    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.319432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.319418    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.319741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.319522    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wmz\" (UniqueName: \"kubernetes.io/projected/05df8544-fb87-4486-baa5-e2d107f355ee-kube-api-access-k5wmz\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.320123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.320097    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.321932 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.321907    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.340682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.340645    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wmz\" (UniqueName: \"kubernetes.io/projected/05df8544-fb87-4486-baa5-e2d107f355ee-kube-api-access-k5wmz\") pod \"maas-api-revoke-sub-b847c1ab-dhjnw\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.429192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.429147    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:41.682989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.682944    4519 generic.go:356] "Generic (PLEG): container finished" podID="518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" containerID="88363467deb7b23ce6a67db8c3ebf1604a2063e3a3f2a2870846764be752068c" exitCode=0
Oct 30 19:51:41.683262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.683149    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz" event={"ID":"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9","Type":"ContainerDied","Data":"88363467deb7b23ce6a67db8c3ebf1604a2063e3a3f2a2870846764be752068c"}
Oct 30 19:51:41.687844 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.687812    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9" event={"ID":"02f383ca-df7f-4185-8aac-d6f3b12dd2c2","Type":"ContainerStarted","Data":"8dc70d0691f73f87dfbc425cb04796d576145beae884aed011d232e1273b3d2c"}
Oct 30 19:51:41.688013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.687868    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9" event={"ID":"02f383ca-df7f-4185-8aac-d6f3b12dd2c2","Type":"ContainerStarted","Data":"eaa83705d78b582e541b94ba09c085e4402fa25eacb4f4173b249662eb2ec621"}
Oct 30 19:51:41.690270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.690237    4519 generic.go:356] "Generic (PLEG): container finished" podID="27956871-d80b-4a5b-bfd5-3bbd0d5b6968" containerID="d25592d11101637f79b979a6e5eb7bab1310eea72332e25dde98569ef01bb74c" exitCode=0
Oct 30 19:51:41.690377 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.690311    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r" event={"ID":"27956871-d80b-4a5b-bfd5-3bbd0d5b6968","Type":"ContainerDied","Data":"d25592d11101637f79b979a6e5eb7bab1310eea72332e25dde98569ef01bb74c"}
Oct 30 19:51:41.741529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.741441    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9" podStartSLOduration=1.7412192370000001 podStartE2EDuration="1.741219237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:41.731368584 +0000 UTC m=+1026.232072309" watchObservedRunningTime="2026-10-30 19:51:41.741219237 +0000 UTC m=+1026.241923243"
Oct 30 19:51:41.807772 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:41.807726    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05df8544_fb87_4486_baa5_e2d107f355ee.slice/crio-5d37e20a491a8ac8ba4f07d1a6ca1fe274138647627ee926ea49a0112e806152 WatchSource:0}: Error finding container 5d37e20a491a8ac8ba4f07d1a6ca1fe274138647627ee926ea49a0112e806152: Status 404 returned error can't find the container with id 5d37e20a491a8ac8ba4f07d1a6ca1fe274138647627ee926ea49a0112e806152
Oct 30 19:51:41.809154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.809115    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"]
Oct 30 19:51:41.857280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:41.857241    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:42.032274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.032238    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-xks-ca\") pod \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") "
Oct 30 19:51:42.032766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.032374    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-openshift-ca\") pod \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") "
Oct 30 19:51:42.032766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.032547    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-kube-api-access-7527g\" (UniqueName: \"kubernetes.io/projected/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-kube-api-access-7527g\") pod \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\" (UID: \"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9\") "
Oct 30 19:51:42.032900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.032784    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-openshift-ca" pod "518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" (UID: "518e2010-cfb2-4726-a42f-4f0cfb7ff3a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:42.033123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.033096    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:42.048444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.048388    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-kube-api-access-7527g" pod "518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" (UID: "518e2010-cfb2-4726-a42f-4f0cfb7ff3a9"). InnerVolumeSpecName "kube-api-access-7527g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:42.049735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.049700    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-xks-ca" pod "518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" (UID: "518e2010-cfb2-4726-a42f-4f0cfb7ff3a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:42.134559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.134409    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7527g\" (UniqueName: \"kubernetes.io/projected/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-kube-api-access-7527g\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:42.134559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.134534    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:42.268669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.268622    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"]
Oct 30 19:51:42.269403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.269378    4519 cpu_manager.go:414] "containerMap: removing container" podUID="518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" containerName="revoke-keys"
Oct 30 19:51:42.269518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.269403    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" containerName="revoke-keys"
Oct 30 19:51:42.269592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.269578    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" containerName="revoke-keys"
Oct 30 19:51:42.274215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.274186    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.308379 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.308310    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"]
Oct 30 19:51:42.337277 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337230    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337287    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337384    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337474    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wdb\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-kube-api-access-m8wdb\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337598    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337681 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337652    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337784 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337762    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337861 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337844    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.337931 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.337917    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.440664 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.440619    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.440896 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.440716    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.440896 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.440870    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.441031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.440937    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.441031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.440970    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.441154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.441045    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wdb\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-kube-api-access-m8wdb\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.441246 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.441220    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.441388 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.441283    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.441388 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.441347    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.442686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.442652    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.442943 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.442920    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.443184 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.443162    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.443417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.443398    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.445832 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.445024    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.447325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.447274    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.452487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.449108    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.465259 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.465224    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.465800 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.465770    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wdb\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-kube-api-access-m8wdb\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.611921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.611877    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:42.715629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.712766    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw" event={"ID":"05df8544-fb87-4486-baa5-e2d107f355ee","Type":"ContainerStarted","Data":"86c45e005279d74a2591adbfd763fe2a4cd00519afedf655bd80eba9d727831d"}
Oct 30 19:51:42.715629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.712839    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw" event={"ID":"05df8544-fb87-4486-baa5-e2d107f355ee","Type":"ContainerStarted","Data":"5d37e20a491a8ac8ba4f07d1a6ca1fe274138647627ee926ea49a0112e806152"}
Oct 30 19:51:42.716714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.716678    4519 generic.go:356] "Generic (PLEG): container finished" podID="02f383ca-df7f-4185-8aac-d6f3b12dd2c2" containerID="8dc70d0691f73f87dfbc425cb04796d576145beae884aed011d232e1273b3d2c" exitCode=0
Oct 30 19:51:42.716813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.716756    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9" event={"ID":"02f383ca-df7f-4185-8aac-d6f3b12dd2c2","Type":"ContainerDied","Data":"8dc70d0691f73f87dfbc425cb04796d576145beae884aed011d232e1273b3d2c"}
Oct 30 19:51:42.718541 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.718517    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"
Oct 30 19:51:42.718662 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.718521    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz" event={"ID":"518e2010-cfb2-4726-a42f-4f0cfb7ff3a9","Type":"ContainerDied","Data":"445fff388c06e09f6779c000fc69300126e838b674618cef3a90b2cf89868e49"}
Oct 30 19:51:42.718662 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.718635    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="445fff388c06e09f6779c000fc69300126e838b674618cef3a90b2cf89868e49"
Oct 30 19:51:42.760603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.760529    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw" podStartSLOduration=1.760501531 podStartE2EDuration="1.760501531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:42.752902311 +0000 UTC m=+1027.253606043" watchObservedRunningTime="2026-10-30 19:51:42.760501531 +0000 UTC m=+1027.261205258"
Oct 30 19:51:42.934521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.934490    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:42.947896 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:42.947856    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242639b8_6ae8_4a9f_9744_ddcccb88b59b.slice/crio-b77d1f01bc6f2347b87e44d5850c1cd2b2918c8717ea5d9fc85a3f94b923c91a WatchSource:0}: Error finding container b77d1f01bc6f2347b87e44d5850c1cd2b2918c8717ea5d9fc85a3f94b923c91a: Status 404 returned error can't find the container with id b77d1f01bc6f2347b87e44d5850c1cd2b2918c8717ea5d9fc85a3f94b923c91a
Oct 30 19:51:42.948714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.948686    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-openshift-ca\") pod \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") "
Oct 30 19:51:42.948935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.948907    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-kube-api-access-mbjxk\" (UniqueName: \"kubernetes.io/projected/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-kube-api-access-mbjxk\") pod \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") "
Oct 30 19:51:42.949057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.949038    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-xks-ca\") pod \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\" (UID: \"27956871-d80b-4a5b-bfd5-3bbd0d5b6968\") "
Oct 30 19:51:42.949117 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.949093    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-openshift-ca" pod "27956871-d80b-4a5b-bfd5-3bbd0d5b6968" (UID: "27956871-d80b-4a5b-bfd5-3bbd0d5b6968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:42.950702 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.949729    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:42.951434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.951408    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"]
Oct 30 19:51:42.951573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.951535    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:42.951695 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.951619    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:42.951695 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.951680    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:51:42.952179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.952154    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-xks-ca" pod "27956871-d80b-4a5b-bfd5-3bbd0d5b6968" (UID: "27956871-d80b-4a5b-bfd5-3bbd0d5b6968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:42.952300 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:42.952274    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-kube-api-access-mbjxk" pod "27956871-d80b-4a5b-bfd5-3bbd0d5b6968" (UID: "27956871-d80b-4a5b-bfd5-3bbd0d5b6968"). InnerVolumeSpecName "kube-api-access-mbjxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:43.051261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.051217    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjxk\" (UniqueName: \"kubernetes.io/projected/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-kube-api-access-mbjxk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:43.051261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.051259    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27956871-d80b-4a5b-bfd5-3bbd0d5b6968-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:43.730873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.730832    4519 generic.go:356] "Generic (PLEG): container finished" podID="05df8544-fb87-4486-baa5-e2d107f355ee" containerID="86c45e005279d74a2591adbfd763fe2a4cd00519afedf655bd80eba9d727831d" exitCode=0
Oct 30 19:51:43.731106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.730918    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw" event={"ID":"05df8544-fb87-4486-baa5-e2d107f355ee","Type":"ContainerDied","Data":"86c45e005279d74a2591adbfd763fe2a4cd00519afedf655bd80eba9d727831d"}
Oct 30 19:51:43.733105 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.733071    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" event={"ID":"242639b8-6ae8-4a9f-9744-ddcccb88b59b","Type":"ContainerStarted","Data":"17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359"}
Oct 30 19:51:43.733282 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.733137    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" event={"ID":"242639b8-6ae8-4a9f-9744-ddcccb88b59b","Type":"ContainerStarted","Data":"b77d1f01bc6f2347b87e44d5850c1cd2b2918c8717ea5d9fc85a3f94b923c91a"}
Oct 30 19:51:43.735064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.735034    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"
Oct 30 19:51:43.735190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.735064    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r" event={"ID":"27956871-d80b-4a5b-bfd5-3bbd0d5b6968","Type":"ContainerDied","Data":"46f1ca7e3097037cfa695d2adc95c2bdcce19b4cdf3a8a42dab96346d2972b3b"}
Oct 30 19:51:43.735190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:43.735109    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46f1ca7e3097037cfa695d2adc95c2bdcce19b4cdf3a8a42dab96346d2972b3b"
Oct 30 19:51:44.012891 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.012756    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:44.014389 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.014320    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podStartSLOduration=2.014299731 podStartE2EDuration="2.014299731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:43.846196388 +0000 UTC m=+1028.346900113" watchObservedRunningTime="2026-10-30 19:51:44.014299731 +0000 UTC m=+1028.515003457"
Oct 30 19:51:44.091007 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.090951    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-kube-api-access-2mp5b\" (UniqueName: \"kubernetes.io/projected/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-kube-api-access-2mp5b\") pod \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") "
Oct 30 19:51:44.091770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.091743    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-openshift-ca\") pod \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") "
Oct 30 19:51:44.092083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.092055    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-xks-ca\") pod \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\" (UID: \"02f383ca-df7f-4185-8aac-d6f3b12dd2c2\") "
Oct 30 19:51:44.092754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.092729    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-openshift-ca" pod "02f383ca-df7f-4185-8aac-d6f3b12dd2c2" (UID: "02f383ca-df7f-4185-8aac-d6f3b12dd2c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:44.093114 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.093092    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:44.098618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.098586    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-kube-api-access-2mp5b" pod "02f383ca-df7f-4185-8aac-d6f3b12dd2c2" (UID: "02f383ca-df7f-4185-8aac-d6f3b12dd2c2"). InnerVolumeSpecName "kube-api-access-2mp5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:44.100511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.100438    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-xks-ca" pod "02f383ca-df7f-4185-8aac-d6f3b12dd2c2" (UID: "02f383ca-df7f-4185-8aac-d6f3b12dd2c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:44.194218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.194131    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:44.194406 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.194392    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mp5b\" (UniqueName: \"kubernetes.io/projected/02f383ca-df7f-4185-8aac-d6f3b12dd2c2-kube-api-access-2mp5b\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:44.612808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.612722    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:44.614281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.614246    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:51:44.614448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.614307    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:51:44.745009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.744976    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"
Oct 30 19:51:44.745203 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.745008    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9" event={"ID":"02f383ca-df7f-4185-8aac-d6f3b12dd2c2","Type":"ContainerDied","Data":"eaa83705d78b582e541b94ba09c085e4402fa25eacb4f4173b249662eb2ec621"}
Oct 30 19:51:44.745203 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:44.745067    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaa83705d78b582e541b94ba09c085e4402fa25eacb4f4173b249662eb2ec621"
Oct 30 19:51:45.025580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.025543    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:45.115885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.115849    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-xks-ca\") pod \"05df8544-fb87-4486-baa5-e2d107f355ee\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") "
Oct 30 19:51:45.116418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.115906    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05df8544-fb87-4486-baa5-e2d107f355ee-kube-api-access-k5wmz\" (UniqueName: \"kubernetes.io/projected/05df8544-fb87-4486-baa5-e2d107f355ee-kube-api-access-k5wmz\") pod \"05df8544-fb87-4486-baa5-e2d107f355ee\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") "
Oct 30 19:51:45.116418 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.116053    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-openshift-ca\") pod \"05df8544-fb87-4486-baa5-e2d107f355ee\" (UID: \"05df8544-fb87-4486-baa5-e2d107f355ee\") "
Oct 30 19:51:45.116846 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.116812    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-openshift-ca" pod "05df8544-fb87-4486-baa5-e2d107f355ee" (UID: "05df8544-fb87-4486-baa5-e2d107f355ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:45.118336 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.118303    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-xks-ca" pod "05df8544-fb87-4486-baa5-e2d107f355ee" (UID: "05df8544-fb87-4486-baa5-e2d107f355ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:45.118727 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.118694    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05df8544-fb87-4486-baa5-e2d107f355ee-kube-api-access-k5wmz" pod "05df8544-fb87-4486-baa5-e2d107f355ee" (UID: "05df8544-fb87-4486-baa5-e2d107f355ee"). InnerVolumeSpecName "kube-api-access-k5wmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:45.194104 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.194061    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"]
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195024    4519 cpu_manager.go:414] "containerMap: removing container" podUID="02f383ca-df7f-4185-8aac-d6f3b12dd2c2" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195044    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f383ca-df7f-4185-8aac-d6f3b12dd2c2" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195091    4519 cpu_manager.go:414] "containerMap: removing container" podUID="27956871-d80b-4a5b-bfd5-3bbd0d5b6968" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195097    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27956871-d80b-4a5b-bfd5-3bbd0d5b6968" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195112    4519 cpu_manager.go:414] "containerMap: removing container" podUID="05df8544-fb87-4486-baa5-e2d107f355ee" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195117    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05df8544-fb87-4486-baa5-e2d107f355ee" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195205    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f383ca-df7f-4185-8aac-d6f3b12dd2c2" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195216    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="27956871-d80b-4a5b-bfd5-3bbd0d5b6968" containerName="revoke-keys"
Oct 30 19:51:45.195745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.195223    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="05df8544-fb87-4486-baa5-e2d107f355ee" containerName="revoke-keys"
Oct 30 19:51:45.199300 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.199275    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.209501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.209454    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"]
Oct 30 19:51:45.217897 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.217856    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0619998a-5e47-4086-8986-728b863e3827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.218106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.218063    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnt9\" (UniqueName: \"kubernetes.io/projected/0619998a-5e47-4086-8986-728b863e3827-kube-api-access-bhnt9\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.218230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.218206    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0619998a-5e47-4086-8986-728b863e3827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.218377 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.218361    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:45.218440 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.218383    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5wmz\" (UniqueName: \"kubernetes.io/projected/05df8544-fb87-4486-baa5-e2d107f355ee-kube-api-access-k5wmz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:45.218440 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.218398    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05df8544-fb87-4486-baa5-e2d107f355ee-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:45.319409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.319314    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnt9\" (UniqueName: \"kubernetes.io/projected/0619998a-5e47-4086-8986-728b863e3827-kube-api-access-bhnt9\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.319409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.319392    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0619998a-5e47-4086-8986-728b863e3827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.319684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.319546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0619998a-5e47-4086-8986-728b863e3827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.320158 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.320136    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0619998a-5e47-4086-8986-728b863e3827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.322042 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.322016    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0619998a-5e47-4086-8986-728b863e3827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.339360 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.339325    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnt9\" (UniqueName: \"kubernetes.io/projected/0619998a-5e47-4086-8986-728b863e3827-kube-api-access-bhnt9\") pod \"maas-api-revoke-sub-a2ce79df-drk68\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.523196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.523154    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:45.613206 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.613124    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:51:45.613206 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.613172    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:51:45.616902 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.616875    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"]
Oct 30 19:51:45.623376 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.623347    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.645369 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.645334    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"]
Oct 30 19:51:45.726177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.726131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.729522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.726499    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.729522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.726632    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrzc2\" (UniqueName: \"kubernetes.io/projected/2a1f4334-51b3-4dfe-b468-3f09846ba9db-kube-api-access-qrzc2\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.798511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.798476    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"
Oct 30 19:51:45.798696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.798507    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw" event={"ID":"05df8544-fb87-4486-baa5-e2d107f355ee","Type":"ContainerDied","Data":"5d37e20a491a8ac8ba4f07d1a6ca1fe274138647627ee926ea49a0112e806152"}
Oct 30 19:51:45.798696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.798566    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d37e20a491a8ac8ba4f07d1a6ca1fe274138647627ee926ea49a0112e806152"
Oct 30 19:51:45.828523 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.828457    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.828731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.828622    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrzc2\" (UniqueName: \"kubernetes.io/projected/2a1f4334-51b3-4dfe-b468-3f09846ba9db-kube-api-access-qrzc2\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.828807 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.828780    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.829851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.829805    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.831514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.831488    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.889079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.887700    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrzc2\" (UniqueName: \"kubernetes.io/projected/2a1f4334-51b3-4dfe-b468-3f09846ba9db-kube-api-access-qrzc2\") pod \"maas-api-revoke-sub-c5e33d43-ctl4c\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:45.891999 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:51:45.891964    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0619998a_5e47_4086_8986_728b863e3827.slice/crio-af38ab5bd12afde1e2fb2e5255b4d599530d4564ee1ddd324ae10b4a53dee3a1 WatchSource:0}: Error finding container af38ab5bd12afde1e2fb2e5255b4d599530d4564ee1ddd324ae10b4a53dee3a1: Status 404 returned error can't find the container with id af38ab5bd12afde1e2fb2e5255b4d599530d4564ee1ddd324ae10b4a53dee3a1
Oct 30 19:51:45.909778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.904863    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"]
Oct 30 19:51:45.996853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:45.996604    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:46.410909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:46.406245    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"]
Oct 30 19:51:46.613452 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:46.613412    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:51:46.613670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:46.613559    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:51:46.811553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:46.811516    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c" event={"ID":"2a1f4334-51b3-4dfe-b468-3f09846ba9db","Type":"ContainerStarted","Data":"709e06ae8149eca3a64681ea6479a8e4e9d689743cacab576fc8ea781b47817d"}
Oct 30 19:51:46.813454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:46.813421    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68" event={"ID":"0619998a-5e47-4086-8986-728b863e3827","Type":"ContainerStarted","Data":"4cb03f210da33dae1e2e38afd144bc09ea40ba7b68d5cba40b652d396e270ab6"}
Oct 30 19:51:46.813609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:46.813494    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68" event={"ID":"0619998a-5e47-4086-8986-728b863e3827","Type":"ContainerStarted","Data":"af38ab5bd12afde1e2fb2e5255b4d599530d4564ee1ddd324ae10b4a53dee3a1"}
Oct 30 19:51:46.886420 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:46.886335    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68" podStartSLOduration=1.886310791 podStartE2EDuration="1.886310791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:46.85497729 +0000 UTC m=+1031.355681014" watchObservedRunningTime="2026-10-30 19:51:46.886310791 +0000 UTC m=+1031.387014516"
Oct 30 19:51:47.612745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:47.612710    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:51:47.613166 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:47.612764    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:51:47.824832 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:47.824796    4519 generic.go:356] "Generic (PLEG): container finished" podID="2a1f4334-51b3-4dfe-b468-3f09846ba9db" containerID="272c9b9c43a41b9512fde245dcd1fff30a36ffacd410ad593d2c9086863c55b5" exitCode=0
Oct 30 19:51:47.825011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:47.824877    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c" event={"ID":"2a1f4334-51b3-4dfe-b468-3f09846ba9db","Type":"ContainerDied","Data":"272c9b9c43a41b9512fde245dcd1fff30a36ffacd410ad593d2c9086863c55b5"}
Oct 30 19:51:47.826655 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:47.826626    4519 generic.go:356] "Generic (PLEG): container finished" podID="0619998a-5e47-4086-8986-728b863e3827" containerID="4cb03f210da33dae1e2e38afd144bc09ea40ba7b68d5cba40b652d396e270ab6" exitCode=0
Oct 30 19:51:47.826777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:47.826671    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68" event={"ID":"0619998a-5e47-4086-8986-728b863e3827","Type":"ContainerDied","Data":"4cb03f210da33dae1e2e38afd144bc09ea40ba7b68d5cba40b652d396e270ab6"}
Oct 30 19:51:48.612565 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:48.612521    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:51:48.612773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:48.612587    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:51:49.133827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.133793    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:49.209961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.209882    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:49.216257 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.216221    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0619998a-5e47-4086-8986-728b863e3827-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0619998a-5e47-4086-8986-728b863e3827-maas-api-xks-ca\") pod \"0619998a-5e47-4086-8986-728b863e3827\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") "
Oct 30 19:51:49.216456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.216436    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0619998a-5e47-4086-8986-728b863e3827-kube-api-access-bhnt9\" (UniqueName: \"kubernetes.io/projected/0619998a-5e47-4086-8986-728b863e3827-kube-api-access-bhnt9\") pod \"0619998a-5e47-4086-8986-728b863e3827\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") "
Oct 30 19:51:49.216555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.216535    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0619998a-5e47-4086-8986-728b863e3827-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0619998a-5e47-4086-8986-728b863e3827-maas-api-openshift-ca\") pod \"0619998a-5e47-4086-8986-728b863e3827\" (UID: \"0619998a-5e47-4086-8986-728b863e3827\") "
Oct 30 19:51:49.217434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.217406    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0619998a-5e47-4086-8986-728b863e3827-maas-api-openshift-ca" pod "0619998a-5e47-4086-8986-728b863e3827" (UID: "0619998a-5e47-4086-8986-728b863e3827"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:49.219700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.219669    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0619998a-5e47-4086-8986-728b863e3827-maas-api-xks-ca" pod "0619998a-5e47-4086-8986-728b863e3827" (UID: "0619998a-5e47-4086-8986-728b863e3827"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:49.220815 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.220316    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0619998a-5e47-4086-8986-728b863e3827-kube-api-access-bhnt9" pod "0619998a-5e47-4086-8986-728b863e3827" (UID: "0619998a-5e47-4086-8986-728b863e3827"). InnerVolumeSpecName "kube-api-access-bhnt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:49.318326 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.318292    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-xks-ca\") pod \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") "
Oct 30 19:51:49.318503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.318363    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-openshift-ca\") pod \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") "
Oct 30 19:51:49.318629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.318616    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1f4334-51b3-4dfe-b468-3f09846ba9db-kube-api-access-qrzc2\" (UniqueName: \"kubernetes.io/projected/2a1f4334-51b3-4dfe-b468-3f09846ba9db-kube-api-access-qrzc2\") pod \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\" (UID: \"2a1f4334-51b3-4dfe-b468-3f09846ba9db\") "
Oct 30 19:51:49.319020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.318993    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-openshift-ca" pod "2a1f4334-51b3-4dfe-b468-3f09846ba9db" (UID: "2a1f4334-51b3-4dfe-b468-3f09846ba9db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:49.319358 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.319334    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhnt9\" (UniqueName: \"kubernetes.io/projected/0619998a-5e47-4086-8986-728b863e3827-kube-api-access-bhnt9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:49.319358 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.319352    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0619998a-5e47-4086-8986-728b863e3827-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:49.319558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.319362    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0619998a-5e47-4086-8986-728b863e3827-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:49.319558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.319372    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:49.320810 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.320784    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-xks-ca" pod "2a1f4334-51b3-4dfe-b468-3f09846ba9db" (UID: "2a1f4334-51b3-4dfe-b468-3f09846ba9db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:49.320901 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.320812    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1f4334-51b3-4dfe-b468-3f09846ba9db-kube-api-access-qrzc2" pod "2a1f4334-51b3-4dfe-b468-3f09846ba9db" (UID: "2a1f4334-51b3-4dfe-b468-3f09846ba9db"). InnerVolumeSpecName "kube-api-access-qrzc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:49.421019 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.420953    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrzc2\" (UniqueName: \"kubernetes.io/projected/2a1f4334-51b3-4dfe-b468-3f09846ba9db-kube-api-access-qrzc2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:49.421019 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.420996    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1f4334-51b3-4dfe-b468-3f09846ba9db-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:51:49.613100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.612997    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:51:49.613100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.613062    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:51:49.902552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.902228    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c" event={"ID":"2a1f4334-51b3-4dfe-b468-3f09846ba9db","Type":"ContainerDied","Data":"709e06ae8149eca3a64681ea6479a8e4e9d689743cacab576fc8ea781b47817d"}
Oct 30 19:51:49.902552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.902323    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"
Oct 30 19:51:49.913498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.911321    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="709e06ae8149eca3a64681ea6479a8e4e9d689743cacab576fc8ea781b47817d"
Oct 30 19:51:49.913498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.912916    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68" event={"ID":"0619998a-5e47-4086-8986-728b863e3827","Type":"ContainerDied","Data":"af38ab5bd12afde1e2fb2e5255b4d599530d4564ee1ddd324ae10b4a53dee3a1"}
Oct 30 19:51:49.913498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.912974    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af38ab5bd12afde1e2fb2e5255b4d599530d4564ee1ddd324ae10b4a53dee3a1"
Oct 30 19:51:49.913498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:49.913047    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"
Oct 30 19:51:50.617493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:50.615016    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:51:50.617493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:50.615106    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:51:51.222494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.221517    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:51.313493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.312918    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:51.371155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.371100    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:51.372680 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.372639    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:51.706621 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.706568    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:51.821493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.820573    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:51.891493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.891350    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:51.972570 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:51.962576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.029493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.028567    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.227594 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.226021    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.227594 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.226858    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:51:52.320493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.314654    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.320493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.315119    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:51:52.386488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.385601    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.401560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.400575    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.401560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.401102    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:51:52.477591 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.477544    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:51:52.491788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.491680    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:51:52.745889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.711298    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.745889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.711967    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:51:52.782509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.779845    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:52.782509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.779913    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.825482 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.825417    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:52.826996 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.826954    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:51:52.898662 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.898615    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:51:52.903446 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:52.903411    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:51:53.100599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.066265    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:51:53.100599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.066988    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:51:53.104490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.101781    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:53.104490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.103028    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:51:53.195862 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.195821    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:53.196200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.196148    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:53.242690 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.242626    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:53.330495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.324568    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:53.405681 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.405567    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:51:53.483686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.483630    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:53.714015 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.713965    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:53.989556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.929413    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:53.989556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:53.963212    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:51:54.148864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.148803    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:54.149097 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.148910    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:54.267034 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.238974    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:54.541394 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.541284    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:51:54.676250 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.544302    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:51:54.677330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.632093    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:54.677629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.645602    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:54.677883 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.677851    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:54.678282 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:51:54.675731    4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.071s"
Oct 30 19:51:54.678516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.678503    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:51:54.678886 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.678753    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:51:54.679607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:54.679587    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:51:55.031524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.002648    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:55.262509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.259748    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:51:55.270530 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.262840    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:55.270530 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.262904    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:55.297623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.280015    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:55.297623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.281658    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:51:55.297623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.282030    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:51:55.332506 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.328633    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:55.332506 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.328709    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:55.636569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.635933    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Oct 30 19:51:55.734807 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.726034    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:55.753635 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.753574    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:51:55.754078 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:55.754042    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:51:56.111987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.111934    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:56.259804 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.259724    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:51:56.292669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.292610    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:56.341495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.318627    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:51:56.403726 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.403604    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:51:56.403993 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.328301    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:51:56.430035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.374609    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:56.434569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.431231    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:56.536697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.536641    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:56.729493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.728494    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:56.729493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.728975    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:51:56.757985 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.757934    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:51:56.809501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:56.805576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:57.129316 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.129199    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:57.282489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.279708    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:57.282489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.280777    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:51:57.333708 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.333636    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:57.395928 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.395820    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:57.413504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.406756    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:51:57.831107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.831049    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:57.865745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.831446    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:57.996518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:57.971703    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:58.042489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.039638    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.202560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.130811    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:58.202560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.130878    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.219554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.215247    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:51:58.223509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.221553    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:58.223509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.221636    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.354426 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.354371    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:51:58.379495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.355654    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.379495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.375214    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.409533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.383559    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:51:58.409533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.393625    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:51:58.409533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.395632    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 19:51:58.409533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.402831    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:51:58.409533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.403832    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 19:51:58.409533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.404385    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded"
Oct 30 19:51:58.417191 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.417140    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.444721 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.444666    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:58.445057 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.445006    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.447648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.447592    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.648559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.647800    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:51:58.648559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.647859    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:51:58.879750 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.879689    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:58.925565 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:58.898075    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:51:59.097738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.096872    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:59.097738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.096937    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.254496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.254358    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.370630 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.370576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.394195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.394144    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.394825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.394792    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:51:59.416915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.395424    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.433079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.419514    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.435048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.419558    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.896492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.895768    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:51:59.900489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.896973    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"]
Oct 30 19:51:59.900489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.897947    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0619998a-5e47-4086-8986-728b863e3827" containerName="revoke-keys"
Oct 30 19:51:59.900489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.897967    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0619998a-5e47-4086-8986-728b863e3827" containerName="revoke-keys"
Oct 30 19:51:59.900489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.897984    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2a1f4334-51b3-4dfe-b468-3f09846ba9db" containerName="revoke-keys"
Oct 30 19:51:59.900489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.897992    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1f4334-51b3-4dfe-b468-3f09846ba9db" containerName="revoke-keys"
Oct 30 19:51:59.900489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.898154    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0619998a-5e47-4086-8986-728b863e3827" containerName="revoke-keys"
Oct 30 19:51:59.900489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.898166    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1f4334-51b3-4dfe-b468-3f09846ba9db" containerName="revoke-keys"
Oct 30 19:51:59.928148 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.903665    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:51:59.928148 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.926738    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:51:59.928148 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.926823    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:59.942273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.931977    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:59.984970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.984923    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcl6j\" (UniqueName: \"kubernetes.io/projected/3da36518-96eb-400f-9cbf-370aad08ea8e-kube-api-access-mcl6j\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:51:59.985218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.985200    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:51:59.985436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:51:59.985417    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.177556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.093486    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcl6j\" (UniqueName: \"kubernetes.io/projected/3da36518-96eb-400f-9cbf-370aad08ea8e-kube-api-access-mcl6j\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.177556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.093547    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.177556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.093696    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.177556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.094912    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.184403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.184359    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:00.185988 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.185927    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:00.204745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.204705    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"]
Oct 30 19:52:00.344494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.333014    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:00.406611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.397082    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.406611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.397776    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcl6j\" (UniqueName: \"kubernetes.io/projected/3da36518-96eb-400f-9cbf-370aad08ea8e-kube-api-access-mcl6j\") pod \"maas-api-revoke-sub-09142759-wqkdw\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.427489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.407839    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:52:00.428353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.428030    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:00.473188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.432259    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:52:00.473188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.432611    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:52:00.473188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.456585    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:00.473188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.461387    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 19:52:00.473188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.469758    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:00.473188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.471903    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:00.584846 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.577091    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:52:00.612698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.612625    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"]
Oct 30 19:52:00.669795 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.668711    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:00.713544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.713433    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"]
Oct 30 19:52:00.904091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.845280    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxdf\" (UniqueName: \"kubernetes.io/projected/2f950276-7c06-4ba2-bf00-3523d9e1002a-kube-api-access-gvxdf\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:00.904091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.845565    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:00.904091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.845651    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:00.934453 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:00.934362    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:52:01.003813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.002811    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:52:01.029280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.029250    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:52:01.034399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.034372    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:52:01.041636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.041578    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:01.045190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.045156    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:01.045690 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.045669    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxdf\" (UniqueName: \"kubernetes.io/projected/2f950276-7c06-4ba2-bf00-3523d9e1002a-kube-api-access-gvxdf\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:01.046024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.046010    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:01.050500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.050453    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:01.056971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.056915    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:01.101740 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.100625    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:52:01.113447 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.111758    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:52:01.115049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.113818    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:52:01.121181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.121156    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:52:01.122992 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.122952    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxdf\" (UniqueName: \"kubernetes.io/projected/2f950276-7c06-4ba2-bf00-3523d9e1002a-kube-api-access-gvxdf\") pod \"maas-api-revoke-sub-80e77105-f6mvp\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:01.228036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.227890    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:01.228036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.227955    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:01.324555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.322545    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"]
Oct 30 19:52:01.339424 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.339389    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:01.408852 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.408815    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:52:01.471559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.469594    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:52:01.558745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.558705    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw" event={"ID":"3da36518-96eb-400f-9cbf-370aad08ea8e","Type":"ContainerStarted","Data":"9c57d48195ddacb6e704593d7523a72aaaaa2eb46ef21123c4c07dd539b748d7"}
Oct 30 19:52:01.753241 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:01.753151    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f950276_7c06_4ba2_bf00_3523d9e1002a.slice/crio-b0ee8f4cd6d52e2cd2bdaa91a4b4a596441c3bf49ad13d0376bdf3e1c0327bee WatchSource:0}: Error finding container b0ee8f4cd6d52e2cd2bdaa91a4b4a596441c3bf49ad13d0376bdf3e1c0327bee: Status 404 returned error can't find the container with id b0ee8f4cd6d52e2cd2bdaa91a4b4a596441c3bf49ad13d0376bdf3e1c0327bee
Oct 30 19:52:01.756948 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:01.756923    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"]
Oct 30 19:52:02.222732 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:52:02.222698    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:52:02.571210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:02.571176    4519 generic.go:356] "Generic (PLEG): container finished" podID="3da36518-96eb-400f-9cbf-370aad08ea8e" containerID="67f6c07596e6ffcfbeab13ef716be411af19c3ab300af1c987ddbba243996823" exitCode=0
Oct 30 19:52:02.571776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:02.571264    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw" event={"ID":"3da36518-96eb-400f-9cbf-370aad08ea8e","Type":"ContainerDied","Data":"67f6c07596e6ffcfbeab13ef716be411af19c3ab300af1c987ddbba243996823"}
Oct 30 19:52:02.573479 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:02.573441    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp" event={"ID":"2f950276-7c06-4ba2-bf00-3523d9e1002a","Type":"ContainerStarted","Data":"cc3b7ff38028bef950dd36692364ad5c2c54837cbb9ea02460015d717636a27c"}
Oct 30 19:52:02.573636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:02.573509    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp" event={"ID":"2f950276-7c06-4ba2-bf00-3523d9e1002a","Type":"ContainerStarted","Data":"b0ee8f4cd6d52e2cd2bdaa91a4b4a596441c3bf49ad13d0376bdf3e1c0327bee"}
Oct 30 19:52:02.684281 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:02.684212    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp" podStartSLOduration=2.684191194 podStartE2EDuration="2.684191194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:02.67217006 +0000 UTC m=+1047.172873785" watchObservedRunningTime="2026-10-30 19:52:02.684191194 +0000 UTC m=+1047.184894920"
Oct 30 19:52:03.585101 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.585062    4519 generic.go:356] "Generic (PLEG): container finished" podID="2f950276-7c06-4ba2-bf00-3523d9e1002a" containerID="cc3b7ff38028bef950dd36692364ad5c2c54837cbb9ea02460015d717636a27c" exitCode=0
Oct 30 19:52:03.585684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.585147    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp" event={"ID":"2f950276-7c06-4ba2-bf00-3523d9e1002a","Type":"ContainerDied","Data":"cc3b7ff38028bef950dd36692364ad5c2c54837cbb9ea02460015d717636a27c"}
Oct 30 19:52:03.774121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.774086    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:03.826921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.826883    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-openshift-ca\") pod \"3da36518-96eb-400f-9cbf-370aad08ea8e\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") "
Oct 30 19:52:03.827111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.827090    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-xks-ca\") pod \"3da36518-96eb-400f-9cbf-370aad08ea8e\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") "
Oct 30 19:52:03.827205 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.827179    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da36518-96eb-400f-9cbf-370aad08ea8e-kube-api-access-mcl6j\" (UniqueName: \"kubernetes.io/projected/3da36518-96eb-400f-9cbf-370aad08ea8e-kube-api-access-mcl6j\") pod \"3da36518-96eb-400f-9cbf-370aad08ea8e\" (UID: \"3da36518-96eb-400f-9cbf-370aad08ea8e\") "
Oct 30 19:52:03.827319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.827293    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-openshift-ca" pod "3da36518-96eb-400f-9cbf-370aad08ea8e" (UID: "3da36518-96eb-400f-9cbf-370aad08ea8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:03.830350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.830099    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-xks-ca" pod "3da36518-96eb-400f-9cbf-370aad08ea8e" (UID: "3da36518-96eb-400f-9cbf-370aad08ea8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:03.830350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.830259    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da36518-96eb-400f-9cbf-370aad08ea8e-kube-api-access-mcl6j" pod "3da36518-96eb-400f-9cbf-370aad08ea8e" (UID: "3da36518-96eb-400f-9cbf-370aad08ea8e"). InnerVolumeSpecName "kube-api-access-mcl6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:03.832179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.832150    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:03.832306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.832182    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da36518-96eb-400f-9cbf-370aad08ea8e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:03.832306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:03.832197    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcl6j\" (UniqueName: \"kubernetes.io/projected/3da36518-96eb-400f-9cbf-370aad08ea8e-kube-api-access-mcl6j\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:04.596956 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.596918    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"
Oct 30 19:52:04.596956 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.596948    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw" event={"ID":"3da36518-96eb-400f-9cbf-370aad08ea8e","Type":"ContainerDied","Data":"9c57d48195ddacb6e704593d7523a72aaaaa2eb46ef21123c4c07dd539b748d7"}
Oct 30 19:52:04.597557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.597000    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c57d48195ddacb6e704593d7523a72aaaaa2eb46ef21123c4c07dd539b748d7"
Oct 30 19:52:04.766799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.766766    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:04.847146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.847025    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-xks-ca\") pod \"2f950276-7c06-4ba2-bf00-3523d9e1002a\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") "
Oct 30 19:52:04.847146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.847135    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f950276-7c06-4ba2-bf00-3523d9e1002a-kube-api-access-gvxdf\" (UniqueName: \"kubernetes.io/projected/2f950276-7c06-4ba2-bf00-3523d9e1002a-kube-api-access-gvxdf\") pod \"2f950276-7c06-4ba2-bf00-3523d9e1002a\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") "
Oct 30 19:52:04.847433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.847205    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-openshift-ca\") pod \"2f950276-7c06-4ba2-bf00-3523d9e1002a\" (UID: \"2f950276-7c06-4ba2-bf00-3523d9e1002a\") "
Oct 30 19:52:04.848720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.848688    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-openshift-ca" pod "2f950276-7c06-4ba2-bf00-3523d9e1002a" (UID: "2f950276-7c06-4ba2-bf00-3523d9e1002a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:04.852804 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.852765    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f950276-7c06-4ba2-bf00-3523d9e1002a-kube-api-access-gvxdf" pod "2f950276-7c06-4ba2-bf00-3523d9e1002a" (UID: "2f950276-7c06-4ba2-bf00-3523d9e1002a"). InnerVolumeSpecName "kube-api-access-gvxdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:04.871657 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.871610    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-xks-ca" pod "2f950276-7c06-4ba2-bf00-3523d9e1002a" (UID: "2f950276-7c06-4ba2-bf00-3523d9e1002a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:04.949973 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.949937    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:04.949973 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.949974    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxdf\" (UniqueName: \"kubernetes.io/projected/2f950276-7c06-4ba2-bf00-3523d9e1002a-kube-api-access-gvxdf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:04.949973 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:04.949984    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950276-7c06-4ba2-bf00-3523d9e1002a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:05.615044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:05.615004    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"
Oct 30 19:52:05.619306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:05.619267    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp" event={"ID":"2f950276-7c06-4ba2-bf00-3523d9e1002a","Type":"ContainerDied","Data":"b0ee8f4cd6d52e2cd2bdaa91a4b4a596441c3bf49ad13d0376bdf3e1c0327bee"}
Oct 30 19:52:05.619536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:05.619335    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0ee8f4cd6d52e2cd2bdaa91a4b4a596441c3bf49ad13d0376bdf3e1c0327bee"
Oct 30 19:52:09.565267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.565216    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"]
Oct 30 19:52:09.566119 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.566094    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3da36518-96eb-400f-9cbf-370aad08ea8e" containerName="revoke-keys"
Oct 30 19:52:09.566230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.566120    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da36518-96eb-400f-9cbf-370aad08ea8e" containerName="revoke-keys"
Oct 30 19:52:09.566230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.566166    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2f950276-7c06-4ba2-bf00-3523d9e1002a" containerName="revoke-keys"
Oct 30 19:52:09.566230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.566175    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f950276-7c06-4ba2-bf00-3523d9e1002a" containerName="revoke-keys"
Oct 30 19:52:09.566428 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.566406    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f950276-7c06-4ba2-bf00-3523d9e1002a" containerName="revoke-keys"
Oct 30 19:52:09.566600 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.566437    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da36518-96eb-400f-9cbf-370aad08ea8e" containerName="revoke-keys"
Oct 30 19:52:09.571367 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.571327    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.586952 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.586917    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:09.587673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.587021    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:52:09.593231 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.593190    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"]
Oct 30 19:52:09.728588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.728506    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhv4c\" (UniqueName: \"kubernetes.io/projected/9ee27653-d254-4603-baee-c5e24346fc72-kube-api-access-xhv4c\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.728808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.728707    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.728917 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.728887    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.830795 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.830664    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhv4c\" (UniqueName: \"kubernetes.io/projected/9ee27653-d254-4603-baee-c5e24346fc72-kube-api-access-xhv4c\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.830995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.830907    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.830995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.830984    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.831827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.831802    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.833563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.833537    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.854331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.854284    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhv4c\" (UniqueName: \"kubernetes.io/projected/9ee27653-d254-4603-baee-c5e24346fc72-kube-api-access-xhv4c\") pod \"maas-api-revoke-sub-baa470dd-c74bb\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:09.888859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:09.888817    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:10.205022 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:10.204975    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee27653_d254_4603_baee_c5e24346fc72.slice/crio-d12aa855d92cf3931bf91831185f396530c5e4dff5d2f54326f851009ff2cd9e WatchSource:0}: Error finding container d12aa855d92cf3931bf91831185f396530c5e4dff5d2f54326f851009ff2cd9e: Status 404 returned error can't find the container with id d12aa855d92cf3931bf91831185f396530c5e4dff5d2f54326f851009ff2cd9e
Oct 30 19:52:10.207249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:10.207216    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"]
Oct 30 19:52:10.669747 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:10.669708    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb" event={"ID":"9ee27653-d254-4603-baee-c5e24346fc72","Type":"ContainerStarted","Data":"be21b0f5678cf669d6079f5a4046c81d5cc5622b851fd9f685246bb6fb99c9b7"}
Oct 30 19:52:10.670162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:10.669774    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb" event={"ID":"9ee27653-d254-4603-baee-c5e24346fc72","Type":"ContainerStarted","Data":"d12aa855d92cf3931bf91831185f396530c5e4dff5d2f54326f851009ff2cd9e"}
Oct 30 19:52:10.713899 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:10.713775    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb" podStartSLOduration=1.7137573590000001 podStartE2EDuration="1.713757359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:10.711396146 +0000 UTC m=+1055.212099870" watchObservedRunningTime="2026-10-30 19:52:10.713757359 +0000 UTC m=+1055.214461083"
Oct 30 19:52:11.685076 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:11.685031    4519 generic.go:356] "Generic (PLEG): container finished" podID="9ee27653-d254-4603-baee-c5e24346fc72" containerID="be21b0f5678cf669d6079f5a4046c81d5cc5622b851fd9f685246bb6fb99c9b7" exitCode=0
Oct 30 19:52:11.685609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:11.685111    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb" event={"ID":"9ee27653-d254-4603-baee-c5e24346fc72","Type":"ContainerDied","Data":"be21b0f5678cf669d6079f5a4046c81d5cc5622b851fd9f685246bb6fb99c9b7"}
Oct 30 19:52:12.356395 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.356354    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"]
Oct 30 19:52:12.361838 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.361787    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.373913 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.373882    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.373935    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.373956    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374220 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.374105    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrctr\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-kube-api-access-zrctr\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374220 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.374136    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.374345    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.374511    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.374566    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.374702 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.374680    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.389544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.389507    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"]
Oct 30 19:52:12.476722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.476679    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.476915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.476738    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.476915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.476819    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.476915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.476861    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.476915 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.476883    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477142 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.476952    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrctr\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-kube-api-access-zrctr\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477142 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.476998    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477300 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.477275    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.477192    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477420 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.477398    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477594 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.477574    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.477618    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.477669    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.477760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.477686    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.479711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.479678    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.480033 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.480010    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.497934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.497886    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.498243 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.498028    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrctr\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-kube-api-access-zrctr\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.550433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.550391    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"]
Oct 30 19:52:12.556278 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.556247    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.579555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.579508    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.579758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.579595    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.579758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.579678    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.579758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.579718    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"]
Oct 30 19:52:12.579930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.579781    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.580010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.579989    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.580109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.580087    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.580173 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.580131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.580173 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.580155    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be62cbee-3784-4ec7-905c-f2751cf55f0e-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.580241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.580176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69vq\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-kube-api-access-z69vq\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.677621 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.677586    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:12.681620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.681412    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.681620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.681513    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.681620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.681593    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.681780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.681656    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.681780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.681686    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be62cbee-3784-4ec7-905c-f2751cf55f0e-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.681780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.681711    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69vq\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-kube-api-access-z69vq\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.681960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.681931    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.682033 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.682001    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.682079 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.682069    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.683134 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.683095    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.684761 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.684737    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.685450 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.685428    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be62cbee-3784-4ec7-905c-f2751cf55f0e-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.686712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.686689    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.686913 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.686887    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.689537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.687203    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.690808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.690779    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.705363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.705335    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.706382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.706356    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69vq\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-kube-api-access-z69vq\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.835273 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.835226    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"]
Oct 30 19:52:12.841856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.841822    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.879458 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.879404    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:12.890011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.889376    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.890011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.889452    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.890011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.889537    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.890011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.889669    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.890874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.889708    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvdv\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-kube-api-access-hwvdv\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.891114 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.891090    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.891188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.891130    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.891307 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.891186    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.891307 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.891221    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.915835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.915792    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"]
Oct 30 19:52:12.945754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.945718    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.994335    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-xks-ca\") pod \"9ee27653-d254-4603-baee-c5e24346fc72\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") "
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.994509    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-openshift-ca\") pod \"9ee27653-d254-4603-baee-c5e24346fc72\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") "
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.994731    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee27653-d254-4603-baee-c5e24346fc72-kube-api-access-xhv4c\" (UniqueName: \"kubernetes.io/projected/9ee27653-d254-4603-baee-c5e24346fc72-kube-api-access-xhv4c\") pod \"9ee27653-d254-4603-baee-c5e24346fc72\" (UID: \"9ee27653-d254-4603-baee-c5e24346fc72\") "
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995040    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995075    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995130    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995166    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995230    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995275    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995342    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995514    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.995552    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvdv\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-kube-api-access-hwvdv\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.996196    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-openshift-ca" pod "9ee27653-d254-4603-baee-c5e24346fc72" (UID: "9ee27653-d254-4603-baee-c5e24346fc72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:12.996863 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.996374    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.997849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.997154    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.998623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.998063    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.998623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.998348    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:12.998623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:12.998587    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:13.002148 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.002080    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:13.003016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.002985    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:13.003170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.003148    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee27653-d254-4603-baee-c5e24346fc72-kube-api-access-xhv4c" pod "9ee27653-d254-4603-baee-c5e24346fc72" (UID: "9ee27653-d254-4603-baee-c5e24346fc72"). InnerVolumeSpecName "kube-api-access-xhv4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:13.003802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.003759    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-xks-ca" pod "9ee27653-d254-4603-baee-c5e24346fc72" (UID: "9ee27653-d254-4603-baee-c5e24346fc72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:13.032195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.032155    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:13.052024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.051984    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvdv\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-kube-api-access-hwvdv\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:13.056225 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.056176    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.056367 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.056279    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.056367 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.056327    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.061522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.060331    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"]
Oct 30 19:52:13.097501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.097438    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:13.097501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.097481    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhv4c\" (UniqueName: \"kubernetes.io/projected/9ee27653-d254-4603-baee-c5e24346fc72-kube-api-access-xhv4c\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:13.097501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.097499    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee27653-d254-4603-baee-c5e24346fc72-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:13.172441 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.172393    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:13.298739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.295013    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.298739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.295105    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.298739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.295147    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.313114 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.313081    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"]
Oct 30 19:52:13.378081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.376594    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"]
Oct 30 19:52:13.378081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.377289    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9ee27653-d254-4603-baee-c5e24346fc72" containerName="revoke-keys"
Oct 30 19:52:13.378081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.377303    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee27653-d254-4603-baee-c5e24346fc72" containerName="revoke-keys"
Oct 30 19:52:13.378081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.377427    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee27653-d254-4603-baee-c5e24346fc72" containerName="revoke-keys"
Oct 30 19:52:13.384754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.384719    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.407392 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.405250    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"]
Oct 30 19:52:13.413526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.413485    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d749e-6412-405b-81e1-35591f2e706d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.414028 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.413920    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d749e-6412-405b-81e1-35591f2e706d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.414243 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.414222    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkw6\" (UniqueName: \"kubernetes.io/projected/843d749e-6412-405b-81e1-35591f2e706d-kube-api-access-7rkw6\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.517200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.517151    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d749e-6412-405b-81e1-35591f2e706d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.517435 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.517240    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkw6\" (UniqueName: \"kubernetes.io/projected/843d749e-6412-405b-81e1-35591f2e706d-kube-api-access-7rkw6\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.517682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.517503    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d749e-6412-405b-81e1-35591f2e706d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.518507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.518458    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d749e-6412-405b-81e1-35591f2e706d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.522043 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.521647    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.522154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.522114    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.522232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.522175    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:13.528815 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.527383    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"]
Oct 30 19:52:13.530277 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.530235    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d749e-6412-405b-81e1-35591f2e706d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.539344 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.539274    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkw6\" (UniqueName: \"kubernetes.io/projected/843d749e-6412-405b-81e1-35591f2e706d-kube-api-access-7rkw6\") pod \"maas-api-revoke-sub-92b0b6a2-kln5l\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.713641 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.713606    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:13.724581 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.723346    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" event={"ID":"6b0f7b52-9b85-48a0-bf10-0926ad6db89e","Type":"ContainerStarted","Data":"81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef"}
Oct 30 19:52:13.724581 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.723402    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" event={"ID":"6b0f7b52-9b85-48a0-bf10-0926ad6db89e","Type":"ContainerStarted","Data":"1765a6d5921dba4fdf6eea9238b9cc13b8eafe31e3db4f098580dcd1cada39d1"}
Oct 30 19:52:13.727890 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.726361    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb" event={"ID":"9ee27653-d254-4603-baee-c5e24346fc72","Type":"ContainerDied","Data":"d12aa855d92cf3931bf91831185f396530c5e4dff5d2f54326f851009ff2cd9e"}
Oct 30 19:52:13.727890 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.726416    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d12aa855d92cf3931bf91831185f396530c5e4dff5d2f54326f851009ff2cd9e"
Oct 30 19:52:13.727890 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.726528    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"
Oct 30 19:52:13.735969 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.735780    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" event={"ID":"be62cbee-3784-4ec7-905c-f2751cf55f0e","Type":"ContainerStarted","Data":"b194a079bc053415ea0f76ac9b92d86a965d9c85dc71dde808b3954d2d6627b4"}
Oct 30 19:52:13.736353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.735945    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" event={"ID":"be62cbee-3784-4ec7-905c-f2751cf55f0e","Type":"ContainerStarted","Data":"b30e0acad8840143e5cc53ad518718672464b5019e2bf6f796dd89eae7831763"}
Oct 30 19:52:13.739111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.739079    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" event={"ID":"2a1bce2e-135b-44ff-9cf3-0dba343d26ef","Type":"ContainerStarted","Data":"de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df"}
Oct 30 19:52:13.739343 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.739322    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" event={"ID":"2a1bce2e-135b-44ff-9cf3-0dba343d26ef","Type":"ContainerStarted","Data":"a7cbd0250532cd6f6a0d13d2ad0f1dcbefd655c22daf23177648d317c32fc4c1"}
Oct 30 19:52:13.817277 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.817213    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podStartSLOduration=1.81718943 podStartE2EDuration="1.81718943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:13.779172327 +0000 UTC m=+1058.279876050" watchObservedRunningTime="2026-10-30 19:52:13.81718943 +0000 UTC m=+1058.317893157"
Oct 30 19:52:13.870492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.869726    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podStartSLOduration=1.8696975390000001 podStartE2EDuration="1.869697539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:13.846007189 +0000 UTC m=+1058.346710912" watchObservedRunningTime="2026-10-30 19:52:13.869697539 +0000 UTC m=+1058.370401264"
Oct 30 19:52:13.891491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.880680    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:13.902899 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:13.902097    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podStartSLOduration=1.9020770790000001 podStartE2EDuration="1.902077079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:13.901305268 +0000 UTC m=+1058.402008993" watchObservedRunningTime="2026-10-30 19:52:13.902077079 +0000 UTC m=+1058.402780806"
Oct 30 19:52:14.238673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.226300    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:14.243113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.240781    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:14.243113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.240862    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:14.244798 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.243804    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"]
Oct 30 19:52:14.683366 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.683333    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:14.694598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.691874    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:52:14.694598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.694302    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:52:14.784224 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.784188    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l" event={"ID":"843d749e-6412-405b-81e1-35591f2e706d","Type":"ContainerStarted","Data":"af29dc48a027aae23577ca0e40ea80b54c31b337932dd86edd2d6618307bb436"}
Oct 30 19:52:14.908493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.902398    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:14.908493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:14.902484    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:15.183494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.178055    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:15.183494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.178140    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:15.704303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.704240    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:52:15.704623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.704588    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:52:15.706828 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.706263    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:52:15.942492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.931568    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:15.942492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.931645    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:15.953561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.953522    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l" event={"ID":"843d749e-6412-405b-81e1-35591f2e706d","Type":"ContainerStarted","Data":"af99783110721d657644cfb557cb556449521be67a33d84c6fdd395d2bc15464"}
Oct 30 19:52:15.976841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:15.976796    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:16.021839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.021801    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"]
Oct 30 19:52:16.058837 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.058789    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.106318 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.106274    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.107233 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.107203    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.107582 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.107562    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7p2\" (UniqueName: \"kubernetes.io/projected/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-kube-api-access-4l7p2\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.156517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.142611    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"]
Oct 30 19:52:16.158208 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.158129    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l" podStartSLOduration=3.158106179 podStartE2EDuration="3.158106179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:16.050081218 +0000 UTC m=+1060.550784943" watchObservedRunningTime="2026-10-30 19:52:16.158106179 +0000 UTC m=+1060.658809903"
Oct 30 19:52:16.215763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.215664    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:16.216072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.216024    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:16.221509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.220108    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.221509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.220268    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.221509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.220416    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7p2\" (UniqueName: \"kubernetes.io/projected/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-kube-api-access-4l7p2\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.227093 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.227058    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.297505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.296567    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:16.340207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.340168    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.445521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.442741    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7p2\" (UniqueName: \"kubernetes.io/projected/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-kube-api-access-4l7p2\") pod \"maas-api-revoke-sub-cb59a549-d8fxz\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.445521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.444434    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"]
Oct 30 19:52:16.499033 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.498940    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.529496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.513073    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:16.550071 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.550032    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:16.558238 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.558135    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.574495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.562153    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7stpc\" (UniqueName: \"kubernetes.io/projected/c202ad29-83e2-4a9b-8037-bcb5db160c6c-kube-api-access-7stpc\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.620288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.620218    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.737195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.731397    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.852036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.731608    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.858507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.852561    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7stpc\" (UniqueName: \"kubernetes.io/projected/c202ad29-83e2-4a9b-8037-bcb5db160c6c-kube-api-access-7stpc\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.880495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.842330    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:16.881293 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.866936    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"]
Oct 30 19:52:16.881602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.819628    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:52:16.881804 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.881773    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:52:16.886129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.886081    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:16.886556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.886520    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:16.913669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:16.913309    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:17.095537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.086547    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7stpc\" (UniqueName: \"kubernetes.io/projected/c202ad29-83e2-4a9b-8037-bcb5db160c6c-kube-api-access-7stpc\") pod \"maas-api-revoke-sub-559c2a8f-tztfg\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:17.158063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.157250    4519 generic.go:356] "Generic (PLEG): container finished" podID="843d749e-6412-405b-81e1-35591f2e706d" containerID="af99783110721d657644cfb557cb556449521be67a33d84c6fdd395d2bc15464" exitCode=0
Oct 30 19:52:17.158063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.157401    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l" event={"ID":"843d749e-6412-405b-81e1-35591f2e706d","Type":"ContainerDied","Data":"af99783110721d657644cfb557cb556449521be67a33d84c6fdd395d2bc15464"}
Oct 30 19:52:17.186081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.184596    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:17.186081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.184661    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:17.214361 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.212776    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:17.701716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.687414    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:52:17.701716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.687498    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:52:17.922631 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.903057    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:17.922631 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:17.903133    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:18.530023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.422987    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 19:52:18.530023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.451884    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Oct 30 19:52:18.530023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.471571    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:52:18.616734 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.582729    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:18.616734 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.582804    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:18.767722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.767635    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:18.836491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.836354    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:52:18.836776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.836727    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:52:18.942404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:18.942341    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.086243 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.086204    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:19.086688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.086590    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:19.377524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.377490    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:19.380646 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.380601    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:19.432546 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.410876    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.450333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.450278    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.511717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.471945    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.511717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.488658    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:52:19.766428 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.766376    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:52:19.787735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.787684    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.788405 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.788370    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:52:19.789365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.789331    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:52:19.789879 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.789856    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:52:19.969495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:19.952805    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:52:20.046021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.045966    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:20.149489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.145039    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:20.174613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.174557    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:20.197490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.176773    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:52:20.217945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.217899    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:52:20.249534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.249460    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:20.249839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.249793    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:20.568076 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.546513    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:52:20.646540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.641583    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:52:20.646540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.641668    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:52:20.646540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.641690    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"]
Oct 30 19:52:20.646540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.641764    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:52:20.646540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.641815    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:52:20.646540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.641843    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"]
Oct 30 19:52:20.646540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.641922    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:52:20.723038 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.721091    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:52:20.723038 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.721159    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 19:52:20.729267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.729104    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:52:20.831512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.831451    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz" event={"ID":"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091","Type":"ContainerStarted","Data":"5b2da3344efa156a4795a29db3ab7ba14162a30ad1046c27cf991f5febc8decf"}
Oct 30 19:52:20.836728 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.836686    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg" event={"ID":"c202ad29-83e2-4a9b-8037-bcb5db160c6c","Type":"ContainerStarted","Data":"a5d23f13787088a166881d73e4139ee8cec2ce4fd30a03983f24fd00a570e325"}
Oct 30 19:52:20.938023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.937986    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:20.938179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:20.938050    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:21.025895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.025558    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:21.048218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.048174    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843d749e-6412-405b-81e1-35591f2e706d-kube-api-access-7rkw6\" (UniqueName: \"kubernetes.io/projected/843d749e-6412-405b-81e1-35591f2e706d-kube-api-access-7rkw6\") pod \"843d749e-6412-405b-81e1-35591f2e706d\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") "
Oct 30 19:52:21.048218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.048237    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843d749e-6412-405b-81e1-35591f2e706d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d749e-6412-405b-81e1-35591f2e706d-maas-api-openshift-ca\") pod \"843d749e-6412-405b-81e1-35591f2e706d\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") "
Oct 30 19:52:21.048534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.048398    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/843d749e-6412-405b-81e1-35591f2e706d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d749e-6412-405b-81e1-35591f2e706d-maas-api-xks-ca\") pod \"843d749e-6412-405b-81e1-35591f2e706d\" (UID: \"843d749e-6412-405b-81e1-35591f2e706d\") "
Oct 30 19:52:21.051919 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.051876    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843d749e-6412-405b-81e1-35591f2e706d-maas-api-openshift-ca" pod "843d749e-6412-405b-81e1-35591f2e706d" (UID: "843d749e-6412-405b-81e1-35591f2e706d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:21.054154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.054110    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843d749e-6412-405b-81e1-35591f2e706d-maas-api-xks-ca" pod "843d749e-6412-405b-81e1-35591f2e706d" (UID: "843d749e-6412-405b-81e1-35591f2e706d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:21.055034 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.055006    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843d749e-6412-405b-81e1-35591f2e706d-kube-api-access-7rkw6" pod "843d749e-6412-405b-81e1-35591f2e706d" (UID: "843d749e-6412-405b-81e1-35591f2e706d"). InnerVolumeSpecName "kube-api-access-7rkw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:21.165629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.165595    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rkw6\" (UniqueName: \"kubernetes.io/projected/843d749e-6412-405b-81e1-35591f2e706d-kube-api-access-7rkw6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:21.165849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.165835    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843d749e-6412-405b-81e1-35591f2e706d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:21.165934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.165923    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843d749e-6412-405b-81e1-35591f2e706d-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:21.175456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.175421    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:21.175773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.175730    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:21.892692 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.892646    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:52:21.893430 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.893395    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 19:52:21.999283 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.999243    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l" event={"ID":"843d749e-6412-405b-81e1-35591f2e706d","Type":"ContainerDied","Data":"af29dc48a027aae23577ca0e40ea80b54c31b337932dd86edd2d6618307bb436"}
Oct 30 19:52:21.999580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.999560    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af29dc48a027aae23577ca0e40ea80b54c31b337932dd86edd2d6618307bb436"
Oct 30 19:52:21.999811 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:21.999797    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"
Oct 30 19:52:22.015548 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.014023    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg" event={"ID":"c202ad29-83e2-4a9b-8037-bcb5db160c6c","Type":"ContainerStarted","Data":"02053e683df865447c726ca4c3f4da0e85fc2d0d5e8791a52a1877e57287b0c3"}
Oct 30 19:52:22.042038 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.041990    4519 generic.go:356] "Generic (PLEG): container finished" podID="c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" containerID="49d1c06c306cadca028f4f68e64501c071bf4daaec9263c552950efbf6bda031" exitCode=0
Oct 30 19:52:22.042232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.042081    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz" event={"ID":"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091","Type":"ContainerDied","Data":"49d1c06c306cadca028f4f68e64501c071bf4daaec9263c552950efbf6bda031"}
Oct 30 19:52:22.076268 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.076175    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg" podStartSLOduration=6.076154492 podStartE2EDuration="6.076154492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:22.064758757 +0000 UTC m=+1066.565462482" watchObservedRunningTime="2026-10-30 19:52:22.076154492 +0000 UTC m=+1066.576858218"
Oct 30 19:52:22.107294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.105458    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:22.107294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.107111    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:22.108597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.108536    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:52:22.173397 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.173354    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:52:22.173631 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.173432    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 19:52:22.888621 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.888582    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:22.889528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.889495    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:22.890694 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:22.890670    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:52:23.055298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.055261    4519 generic.go:356] "Generic (PLEG): container finished" podID="c202ad29-83e2-4a9b-8037-bcb5db160c6c" containerID="02053e683df865447c726ca4c3f4da0e85fc2d0d5e8791a52a1877e57287b0c3" exitCode=0
Oct 30 19:52:23.055839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.055340    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg" event={"ID":"c202ad29-83e2-4a9b-8037-bcb5db160c6c","Type":"ContainerDied","Data":"02053e683df865447c726ca4c3f4da0e85fc2d0d5e8791a52a1877e57287b0c3"}
Oct 30 19:52:23.179301 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.179264    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:23.181056 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.181024    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:23.182061 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.182038    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 19:52:23.262342 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.262305    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:23.367492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.361173    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-openshift-ca\") pod \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") "
Oct 30 19:52:23.367492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.361354    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-xks-ca\") pod \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") "
Oct 30 19:52:23.367492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.361578    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-kube-api-access-4l7p2\" (UniqueName: \"kubernetes.io/projected/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-kube-api-access-4l7p2\") pod \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\" (UID: \"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091\") "
Oct 30 19:52:23.367492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.363276    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-openshift-ca" pod "c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" (UID: "c3c2bb5e-7ebc-4605-9f3b-519c4d72f091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:23.385295 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.385249    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-kube-api-access-4l7p2" pod "c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" (UID: "c3c2bb5e-7ebc-4605-9f3b-519c4d72f091"). InnerVolumeSpecName "kube-api-access-4l7p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:23.388422 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.388362    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-xks-ca" pod "c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" (UID: "c3c2bb5e-7ebc-4605-9f3b-519c4d72f091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:23.464026 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.463983    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:23.464026 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.464026    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l7p2\" (UniqueName: \"kubernetes.io/projected/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-kube-api-access-4l7p2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:23.464252 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.464043    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:23.934566 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.934527    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"]
Oct 30 19:52:23.934795 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:23.934765    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" containerID="cri-o://67ca2b70de02ca50190765b5761b4bf1841734db240fd4efb1c155491208cd2b" gracePeriod=30
Oct 30 19:52:24.085544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.083531    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"
Oct 30 19:52:24.085544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.083766    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz" event={"ID":"c3c2bb5e-7ebc-4605-9f3b-519c4d72f091","Type":"ContainerDied","Data":"5b2da3344efa156a4795a29db3ab7ba14162a30ad1046c27cf991f5febc8decf"}
Oct 30 19:52:24.085544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.083868    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b2da3344efa156a4795a29db3ab7ba14162a30ad1046c27cf991f5febc8decf"
Oct 30 19:52:24.271065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.271030    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:24.385851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.385809    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-xks-ca\") pod \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") "
Oct 30 19:52:24.386074 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.386054    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c202ad29-83e2-4a9b-8037-bcb5db160c6c-kube-api-access-7stpc\" (UniqueName: \"kubernetes.io/projected/c202ad29-83e2-4a9b-8037-bcb5db160c6c-kube-api-access-7stpc\") pod \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") "
Oct 30 19:52:24.386275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.386252    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-openshift-ca\") pod \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\" (UID: \"c202ad29-83e2-4a9b-8037-bcb5db160c6c\") "
Oct 30 19:52:24.386940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.386652    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-openshift-ca" pod "c202ad29-83e2-4a9b-8037-bcb5db160c6c" (UID: "c202ad29-83e2-4a9b-8037-bcb5db160c6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:24.386940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.386912    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:24.388922 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.388890    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c202ad29-83e2-4a9b-8037-bcb5db160c6c-kube-api-access-7stpc" pod "c202ad29-83e2-4a9b-8037-bcb5db160c6c" (UID: "c202ad29-83e2-4a9b-8037-bcb5db160c6c"). InnerVolumeSpecName "kube-api-access-7stpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:24.389013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.388978    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-xks-ca" pod "c202ad29-83e2-4a9b-8037-bcb5db160c6c" (UID: "c202ad29-83e2-4a9b-8037-bcb5db160c6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:24.488592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.488498    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202ad29-83e2-4a9b-8037-bcb5db160c6c-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:24.488592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.488529    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7stpc\" (UniqueName: \"kubernetes.io/projected/c202ad29-83e2-4a9b-8037-bcb5db160c6c-kube-api-access-7stpc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:24.518793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.518753    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"]
Oct 30 19:52:24.519510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519489    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" containerName="revoke-keys"
Oct 30 19:52:24.519510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519510    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" containerName="revoke-keys"
Oct 30 19:52:24.519697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519530    4519 cpu_manager.go:414] "containerMap: removing container" podUID="843d749e-6412-405b-81e1-35591f2e706d" containerName="revoke-keys"
Oct 30 19:52:24.519697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519537    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843d749e-6412-405b-81e1-35591f2e706d" containerName="revoke-keys"
Oct 30 19:52:24.519697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519581    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c202ad29-83e2-4a9b-8037-bcb5db160c6c" containerName="revoke-keys"
Oct 30 19:52:24.519697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519589    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c202ad29-83e2-4a9b-8037-bcb5db160c6c" containerName="revoke-keys"
Oct 30 19:52:24.519874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519770    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" containerName="revoke-keys"
Oct 30 19:52:24.519874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519784    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="843d749e-6412-405b-81e1-35591f2e706d" containerName="revoke-keys"
Oct 30 19:52:24.519874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.519801    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c202ad29-83e2-4a9b-8037-bcb5db160c6c" containerName="revoke-keys"
Oct 30 19:52:24.523666 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.523639    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.543571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.543539    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"]
Oct 30 19:52:24.692315 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.692266    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q495\" (UniqueName: \"kubernetes.io/projected/a6b98049-0de6-4d65-9050-a4949007a453-kube-api-access-2q495\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.692531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.692431    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.692610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.692585    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.794311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.794198    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.794561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.794356    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.794639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.794572    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q495\" (UniqueName: \"kubernetes.io/projected/a6b98049-0de6-4d65-9050-a4949007a453-kube-api-access-2q495\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.795191 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.795161    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.797332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.797307    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.814654 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.814619    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q495\" (UniqueName: \"kubernetes.io/projected/a6b98049-0de6-4d65-9050-a4949007a453-kube-api-access-2q495\") pod \"maas-api-revoke-sub-10fc4068-n997c\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:24.843524 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:24.843446    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:25.098979 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:25.098884    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"
Oct 30 19:52:25.098979 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:25.098892    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg" event={"ID":"c202ad29-83e2-4a9b-8037-bcb5db160c6c","Type":"ContainerDied","Data":"a5d23f13787088a166881d73e4139ee8cec2ce4fd30a03983f24fd00a570e325"}
Oct 30 19:52:25.098979 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:25.098961    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d23f13787088a166881d73e4139ee8cec2ce4fd30a03983f24fd00a570e325"
Oct 30 19:52:25.169340 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:25.169286    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b98049_0de6_4d65_9050_a4949007a453.slice/crio-edf75095391a5f07618ac951fdabf40ad9ccbf64e3328da237262e06a1808a08 WatchSource:0}: Error finding container edf75095391a5f07618ac951fdabf40ad9ccbf64e3328da237262e06a1808a08: Status 404 returned error can't find the container with id edf75095391a5f07618ac951fdabf40ad9ccbf64e3328da237262e06a1808a08
Oct 30 19:52:25.172152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:25.172122    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"]
Oct 30 19:52:26.109940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:26.109901    4519 generic.go:356] "Generic (PLEG): container finished" podID="a6b98049-0de6-4d65-9050-a4949007a453" containerID="4f82fe36c230e62716d968d0c6cede75105fe1a469e9185e1ede07cb6aef6f41" exitCode=0
Oct 30 19:52:26.110417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:26.109983    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c" event={"ID":"a6b98049-0de6-4d65-9050-a4949007a453","Type":"ContainerDied","Data":"4f82fe36c230e62716d968d0c6cede75105fe1a469e9185e1ede07cb6aef6f41"}
Oct 30 19:52:26.110417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:26.110037    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c" event={"ID":"a6b98049-0de6-4d65-9050-a4949007a453","Type":"ContainerStarted","Data":"edf75095391a5f07618ac951fdabf40ad9ccbf64e3328da237262e06a1808a08"}
Oct 30 19:52:27.175491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.170005    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"]
Oct 30 19:52:27.195487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.188578    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.241374 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.241329    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcfw\" (UniqueName: \"kubernetes.io/projected/f5d95dd9-8d43-404e-9625-fa688dabef01-kube-api-access-2dcfw\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.241701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.241418    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.241701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.241648    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.280104 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.265584    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"]
Oct 30 19:52:27.343209 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.343152    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.343461 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.343437    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcfw\" (UniqueName: \"kubernetes.io/projected/f5d95dd9-8d43-404e-9625-fa688dabef01-kube-api-access-2dcfw\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.343543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.343521    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.344179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.344151    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.346253 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.346224    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.364532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.364500    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcfw\" (UniqueName: \"kubernetes.io/projected/f5d95dd9-8d43-404e-9625-fa688dabef01-kube-api-access-2dcfw\") pod \"maas-api-revoke-sub-e752e050-4zndq\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:27.406410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.406373    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:27.444379 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.444326    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-xks-ca\") pod \"a6b98049-0de6-4d65-9050-a4949007a453\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") "
Oct 30 19:52:27.444609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.444409    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-openshift-ca\") pod \"a6b98049-0de6-4d65-9050-a4949007a453\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") "
Oct 30 19:52:27.444609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.444445    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b98049-0de6-4d65-9050-a4949007a453-kube-api-access-2q495\" (UniqueName: \"kubernetes.io/projected/a6b98049-0de6-4d65-9050-a4949007a453-kube-api-access-2q495\") pod \"a6b98049-0de6-4d65-9050-a4949007a453\" (UID: \"a6b98049-0de6-4d65-9050-a4949007a453\") "
Oct 30 19:52:27.445157 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.445126    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-openshift-ca" pod "a6b98049-0de6-4d65-9050-a4949007a453" (UID: "a6b98049-0de6-4d65-9050-a4949007a453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:27.445601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.445578    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:27.447910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.447881    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b98049-0de6-4d65-9050-a4949007a453-kube-api-access-2q495" pod "a6b98049-0de6-4d65-9050-a4949007a453" (UID: "a6b98049-0de6-4d65-9050-a4949007a453"). InnerVolumeSpecName "kube-api-access-2q495". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:27.453438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.453394    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-xks-ca" pod "a6b98049-0de6-4d65-9050-a4949007a453" (UID: "a6b98049-0de6-4d65-9050-a4949007a453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:27.547215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.547171    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b98049-0de6-4d65-9050-a4949007a453-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:27.547215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.547210    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q495\" (UniqueName: \"kubernetes.io/projected/a6b98049-0de6-4d65-9050-a4949007a453-kube-api-access-2q495\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:27.603547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:27.602702    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:28.169086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:28.166622    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c" event={"ID":"a6b98049-0de6-4d65-9050-a4949007a453","Type":"ContainerDied","Data":"edf75095391a5f07618ac951fdabf40ad9ccbf64e3328da237262e06a1808a08"}
Oct 30 19:52:28.169086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:28.166692    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf75095391a5f07618ac951fdabf40ad9ccbf64e3328da237262e06a1808a08"
Oct 30 19:52:28.169086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:28.166782    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"
Oct 30 19:52:28.922494 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:28.906545    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d95dd9_8d43_404e_9625_fa688dabef01.slice/crio-c940a848c132d830871c88ffc8a23237fe4df00f952f83c01872d20886bbc96f WatchSource:0}: Error finding container c940a848c132d830871c88ffc8a23237fe4df00f952f83c01872d20886bbc96f: Status 404 returned error can't find the container with id c940a848c132d830871c88ffc8a23237fe4df00f952f83c01872d20886bbc96f
Oct 30 19:52:29.071526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.050722    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:29.158498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.135870    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"]
Oct 30 19:52:29.169541 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.166518    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:52:29.531306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.531257    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:52:29.709493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.685198    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:29.759615 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.759560    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:29.959504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.957353    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:29.984438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:29.984381    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq" event={"ID":"f5d95dd9-8d43-404e-9625-fa688dabef01","Type":"ContainerStarted","Data":"c940a848c132d830871c88ffc8a23237fe4df00f952f83c01872d20886bbc96f"}
Oct 30 19:52:30.033350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.033307    4519 generic.go:356] "Generic (PLEG): container finished" podID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerID="67ca2b70de02ca50190765b5761b4bf1841734db240fd4efb1c155491208cd2b" exitCode=0
Oct 30 19:52:30.068095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.033642    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" event={"ID":"46e6714c-8a9c-4eb6-9495-2bb6f35e270f","Type":"ContainerDied","Data":"67ca2b70de02ca50190765b5761b4bf1841734db240fd4efb1c155491208cd2b"}
Oct 30 19:52:30.068397 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.067913    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:52:30.075492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.069243    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:52:30.176494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.174605    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.176494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.175417    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:52:30.309495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.298583    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:30.309495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.298655    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.309495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.298975    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.309495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.299325    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:52:30.514503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.491155    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.525526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.523162    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.728514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.724215    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.741833 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.724371    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:52:30.783393 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.758845    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:52:30.783723 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.783690    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:52:30.881626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.858806    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:52:30.881626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.859167    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:52:30.928444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.928394    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.961756 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.961703    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.984495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:30.972903    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:52:31.098700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.098570    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:52:31.281245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.280621    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.303916 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.303870    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:52:31.339495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.315110    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:52:31.370557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.316772    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:52:31.565501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.565443    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:52:31.595600 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.523765    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.599505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.596669    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:52:31.782678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.782615    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.827508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.788876    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.892659 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.892608    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.967508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:31.967249    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:52:32.104495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.092933    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nxd25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:52:32.104495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.092997    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 19:52:32.280936 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.280814    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq" event={"ID":"f5d95dd9-8d43-404e-9625-fa688dabef01","Type":"ContainerStarted","Data":"05da1f7d3bed812b59d65e0422990f6c6978156c083391368cf452992891de45"}
Oct 30 19:52:32.301202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.300780    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:52:32.315497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.302140    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:32.316329 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.316277    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:32.402797 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.386564    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:32.521494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.518118    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:32.563554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.556553    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:32.563554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.554668    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:52:32.597606 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.596392    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:32.818557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.802863    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:32.818557 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.804348    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:52:32.917907 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.897084    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.058532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:32.979934    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:52:33.321495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.321403    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.334355 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.334304    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.335141 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.335111    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:52:33.366677 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.366620    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.425398 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.409319    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25" event={"ID":"46e6714c-8a9c-4eb6-9495-2bb6f35e270f","Type":"ContainerDied","Data":"c8185492fe4ff0f47de4bd089c511692017e79a470a3b5f2e0e50c2483ee0c58"}
Oct 30 19:52:33.428691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.428652    4519 scope.go:122] "RemoveContainer" containerID="67ca2b70de02ca50190765b5761b4bf1841734db240fd4efb1c155491208cd2b"
Oct 30 19:52:33.429129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.424731    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"
Oct 30 19:52:33.498494 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:52:33.482693    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:52:33.498494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.484289    4519 generic.go:356] "Generic (PLEG): container finished" podID="f5d95dd9-8d43-404e-9625-fa688dabef01" containerID="05da1f7d3bed812b59d65e0422990f6c6978156c083391368cf452992891de45" exitCode=0
Oct 30 19:52:33.498494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.484440    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq" event={"ID":"f5d95dd9-8d43-404e-9625-fa688dabef01","Type":"ContainerDied","Data":"05da1f7d3bed812b59d65e0422990f6c6978156c083391368cf452992891de45"}
Oct 30 19:52:33.580648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.563317    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-data\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.580648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.563508    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-podinfo\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.580648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.563560    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-certs\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.580648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.563612    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istiod-ca-cert\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.580648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.563708    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-credential-socket\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.606441 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.606355    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-credential-socket" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:33.616778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.616723    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.661503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.617759    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-data" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:33.751941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.636311    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-socket\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.752370 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.752341    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-envoy\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.759649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.759592    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-kube-api-access-pldsk\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-kube-api-access-pldsk\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.776713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.776673    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-token\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-token\") pod \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\" (UID: \"46e6714c-8a9c-4eb6-9495-2bb6f35e270f\") "
Oct 30 19:52:33.882560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.797250    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-socket" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:33.901991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.901947    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istiod-ca-cert" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:33.902822 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.902793    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-certs" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:33.906914 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.906861    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:52:33.920025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.906814    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:52:33.920025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.910835    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:33.920025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.910872    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:33.920025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.910888    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:33.920025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.910902    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:33.920025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.910915    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:33.944178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.944122    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-podinfo" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:52:33.951572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.949692    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:52:33.993270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:33.990707    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-envoy" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:34.022882 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.022568    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.030816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.030759    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-kube-api-access-pldsk" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "kube-api-access-pldsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:34.046362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.046323    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:34.046362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.046362    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:34.046362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.046373    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pldsk\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-kube-api-access-pldsk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:34.146560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.080567    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-token" pod "46e6714c-8a9c-4eb6-9495-2bb6f35e270f" (UID: "46e6714c-8a9c-4eb6-9495-2bb6f35e270f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:34.146560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.080680    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.177529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.155079    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e6714c-8a9c-4eb6-9495-2bb6f35e270f-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:34.309495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.289072    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq" podStartSLOduration=7.289050578 podStartE2EDuration="7.289050578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:32.35992157 +0000 UTC m=+1076.860625295" watchObservedRunningTime="2026-10-30 19:52:34.289050578 +0000 UTC m=+1078.789754307"
Oct 30 19:52:34.359499 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.337960    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.359499 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.338376    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.387420 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.378718    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.395067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.393648    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:52:34.667081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.649845    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:34.921895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.921841    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.943495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.925510    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.963653 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:34.955251    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.039730 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.039603    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.110509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.084806    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.338233 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.338125    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:52:35.388707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.387750    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.445095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.445051    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Oct 30 19:52:35.679494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.673573    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:35.679494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.673658    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.686096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.679889    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.965651 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:35.936602    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:36.077345 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.053765    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:52:36.445602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.401104    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:36.752588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.740434    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:36.880503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.878928    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:52:36.955493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.951941    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:52:37.029650 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.962656    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:37.029988 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.029952    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.033689 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.978838    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:52:37.092536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:36.979068    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:37.145017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.126287    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.145017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.093908    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:52:37.145017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.092828    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.249458 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.249377    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:37.251487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.249790    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.433493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.432223    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:37.433493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.432350    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.561573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.539189    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:52:37.561573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.545457    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:52:37.584701 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.584627    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:52:37.608974 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.608909    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:37.609307 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.609276    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.715506 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.712447    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:37.715506 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.712547    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.864044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.848442    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:37.905502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:37.894503    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.047496 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:52:38.027659    4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.337s"
Oct 30 19:52:38.109498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.101639    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:52:38.138554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.114706    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:38.235495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.207046    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.235850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.134978    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.236392 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.135008    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:52:38.236840 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.136318    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:52:38.256495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.144941    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:38.257338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.134314    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.257658 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.217266    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:38.257831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.257799    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.306378 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.306211    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:38.306637 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.306412    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.402759 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.393530    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d95dd9-8d43-404e-9625-fa688dabef01-kube-api-access-2dcfw\" (UniqueName: \"kubernetes.io/projected/f5d95dd9-8d43-404e-9625-fa688dabef01-kube-api-access-2dcfw\") pod \"f5d95dd9-8d43-404e-9625-fa688dabef01\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") "
Oct 30 19:52:38.402759 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.393644    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-xks-ca\") pod \"f5d95dd9-8d43-404e-9625-fa688dabef01\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") "
Oct 30 19:52:38.402759 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.393811    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-openshift-ca\") pod \"f5d95dd9-8d43-404e-9625-fa688dabef01\" (UID: \"f5d95dd9-8d43-404e-9625-fa688dabef01\") "
Oct 30 19:52:38.471864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.417922    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:38.471864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.417993    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.475615 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.475574    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d95dd9-8d43-404e-9625-fa688dabef01-kube-api-access-2dcfw" pod "f5d95dd9-8d43-404e-9625-fa688dabef01" (UID: "f5d95dd9-8d43-404e-9625-fa688dabef01"). InnerVolumeSpecName "kube-api-access-2dcfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:38.627492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.521134    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dcfw\" (UniqueName: \"kubernetes.io/projected/f5d95dd9-8d43-404e-9625-fa688dabef01-kube-api-access-2dcfw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.627492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.542581    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.627492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.551737    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-openshift-ca" pod "f5d95dd9-8d43-404e-9625-fa688dabef01" (UID: "f5d95dd9-8d43-404e-9625-fa688dabef01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:38.627492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.551844    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.627492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.600994    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.705571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.651429    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.705571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.664544    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"]
Oct 30 19:52:38.705571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.664591    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nxd25"]
Oct 30 19:52:38.705571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.664662    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:52:38.705571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.696316    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.705571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.696744    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.713493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.712922    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 19:52:38.713493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.713022    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded"
Oct 30 19:52:38.764498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.760764    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-xks-ca" pod "f5d95dd9-8d43-404e-9625-fa688dabef01" (UID: "f5d95dd9-8d43-404e-9625-fa688dabef01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:38.803494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.801530    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d95dd9-8d43-404e-9625-fa688dabef01-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.882558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.868007    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:52:38.907983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:38.907931    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:52:39.129768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.129648    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:39.238073 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.238026    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:52:39.239221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.239180    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:52:39.261904 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.252182    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:39.276614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.274568    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:39.276913 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.276868    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:39.305512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.291931    4519 trace.go:236] Trace[681092554]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-lpfww" (30-Oct-2026 19:52:37.934) (total time: 1357ms):
Oct 30 19:52:39.305512 ip-10-0-133-26 kubenswrapper[4519]: Trace[681092554]: [1.357125974s] [1.357125974s] END
Oct 30 19:52:39.318874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.308518    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq" event={"ID":"f5d95dd9-8d43-404e-9625-fa688dabef01","Type":"ContainerDied","Data":"c940a848c132d830871c88ffc8a23237fe4df00f952f83c01872d20886bbc96f"}
Oct 30 19:52:39.319173 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.319138    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c940a848c132d830871c88ffc8a23237fe4df00f952f83c01872d20886bbc96f"
Oct 30 19:52:39.324492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.319397    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"
Oct 30 19:52:39.579576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.553801    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:52:39.583769 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.557604    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:39.630487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.628642    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:52:39.814337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.810060    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" path="/var/lib/kubelet/pods/46e6714c-8a9c-4eb6-9495-2bb6f35e270f/volumes"
Oct 30 19:52:39.830572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.830539    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:52:39.872717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.872660    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:39.883488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.877649    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:52:39.919522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.908561    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:39.938417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.938379    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:52:39.956485 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.956381    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:52:39.959398 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:39.957588    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:52:40.014986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.014951    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:52:40.015656 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.015631    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:52:40.015862 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.015847    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:52:40.022597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.022558    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:52:40.249830 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.249782    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7059-ffbe09"]
Oct 30 19:52:40.250918 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.250885    4519 cpu_manager.go:414] "containerMap: removing container" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy"
Oct 30 19:52:40.250918 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.250917    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy"
Oct 30 19:52:40.251095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.250987    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a6b98049-0de6-4d65-9050-a4949007a453" containerName="revoke-keys"
Oct 30 19:52:40.251095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.250996    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b98049-0de6-4d65-9050-a4949007a453" containerName="revoke-keys"
Oct 30 19:52:40.251095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.251011    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f5d95dd9-8d43-404e-9625-fa688dabef01" containerName="revoke-keys"
Oct 30 19:52:40.251095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.251019    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d95dd9-8d43-404e-9625-fa688dabef01" containerName="revoke-keys"
Oct 30 19:52:40.251305 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.251196    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b98049-0de6-4d65-9050-a4949007a453" containerName="revoke-keys"
Oct 30 19:52:40.251305 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.251211    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d95dd9-8d43-404e-9625-fa688dabef01" containerName="revoke-keys"
Oct 30 19:52:40.251305 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.251224    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e6714c-8a9c-4eb6-9495-2bb6f35e270f" containerName="istio-proxy"
Oct 30 19:52:40.257793 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.257742    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:40.278805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.275141    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-brtz2\""
Oct 30 19:52:40.283761 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.283716    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-ffbe09"]
Oct 30 19:52:40.294237 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.294203    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxd26\" (UniqueName: \"kubernetes.io/projected/d0db1977-1f53-4971-ab43-2adbcb17d80b-kube-api-access-wxd26\") pod \"test-curl-7059-ffbe09\" (UID: \"d0db1977-1f53-4971-ab43-2adbcb17d80b\") " pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:40.364383 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.364347    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"]
Oct 30 19:52:40.369758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.369727    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.384769 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.384738    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:52:40.384991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.384792    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:40.391353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.391315    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"]
Oct 30 19:52:40.396713 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.396318    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd26\" (UniqueName: \"kubernetes.io/projected/d0db1977-1f53-4971-ab43-2adbcb17d80b-kube-api-access-wxd26\") pod \"test-curl-7059-ffbe09\" (UID: \"d0db1977-1f53-4971-ab43-2adbcb17d80b\") " pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:40.424875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.424844    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd26\" (UniqueName: \"kubernetes.io/projected/d0db1977-1f53-4971-ab43-2adbcb17d80b-kube-api-access-wxd26\") pod \"test-curl-7059-ffbe09\" (UID: \"d0db1977-1f53-4971-ab43-2adbcb17d80b\") " pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:40.498346 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.498296    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f073363-f521-4176-aae3-7b146033048b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.498346 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.498357    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bxz\" (UniqueName: \"kubernetes.io/projected/9f073363-f521-4176-aae3-7b146033048b-kube-api-access-d4bxz\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.498593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.498554    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f073363-f521-4176-aae3-7b146033048b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.588596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.588457    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:40.600340 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.600296    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f073363-f521-4176-aae3-7b146033048b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.600633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.600613    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f073363-f521-4176-aae3-7b146033048b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.600729 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.600649    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bxz\" (UniqueName: \"kubernetes.io/projected/9f073363-f521-4176-aae3-7b146033048b-kube-api-access-d4bxz\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.601297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.601261    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f073363-f521-4176-aae3-7b146033048b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.603059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.603025    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f073363-f521-4176-aae3-7b146033048b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.621691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.621646    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bxz\" (UniqueName: \"kubernetes.io/projected/9f073363-f521-4176-aae3-7b146033048b-kube-api-access-d4bxz\") pod \"maas-api-revoke-sub-ff20d7e7-f4nfv\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.701527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.701432    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:40.738269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.738228    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"]
Oct 30 19:52:40.745311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.745277    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.768245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.768189    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"]
Oct 30 19:52:40.803812 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.803763    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.804432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.803865    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhjc\" (UniqueName: \"kubernetes.io/projected/161c49f7-8406-4566-9703-7f76c8ed443b-kube-api-access-8mhjc\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.804432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.803985    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.907366 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.907211    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhjc\" (UniqueName: \"kubernetes.io/projected/161c49f7-8406-4566-9703-7f76c8ed443b-kube-api-access-8mhjc\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.907660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.907606    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.907819 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.907764    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.908424 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.908400    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.910894 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.910867    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:40.911990 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:40.911962    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0db1977_1f53_4971_ab43_2adbcb17d80b.slice/crio-875b0395b48cf67ecbac7092bdfa26f1f944c349d9ee96d6849d75c25de07301 WatchSource:0}: Error finding container 875b0395b48cf67ecbac7092bdfa26f1f944c349d9ee96d6849d75c25de07301: Status 404 returned error can't find the container with id 875b0395b48cf67ecbac7092bdfa26f1f944c349d9ee96d6849d75c25de07301
Oct 30 19:52:40.913765 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.913695    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-ffbe09"]
Oct 30 19:52:40.914297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.914279    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:52:40.926965 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:40.926930    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhjc\" (UniqueName: \"kubernetes.io/projected/161c49f7-8406-4566-9703-7f76c8ed443b-kube-api-access-8mhjc\") pod \"maas-api-revoke-sub-4cce0870-rw784\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:41.041544 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:41.041504    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f073363_f521_4176_aae3_7b146033048b.slice/crio-0eaa96604168a41c61228d3c3bd3775c765f99ba3789082713eeb34da626ed34 WatchSource:0}: Error finding container 0eaa96604168a41c61228d3c3bd3775c765f99ba3789082713eeb34da626ed34: Status 404 returned error can't find the container with id 0eaa96604168a41c61228d3c3bd3775c765f99ba3789082713eeb34da626ed34
Oct 30 19:52:41.044514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.044444    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"]
Oct 30 19:52:41.067241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.067200    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:41.335886 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.335847    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-ffbe09" event={"ID":"d0db1977-1f53-4971-ab43-2adbcb17d80b","Type":"ContainerStarted","Data":"875b0395b48cf67ecbac7092bdfa26f1f944c349d9ee96d6849d75c25de07301"}
Oct 30 19:52:41.337673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.337642    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv" event={"ID":"9f073363-f521-4176-aae3-7b146033048b","Type":"ContainerStarted","Data":"0eaa96604168a41c61228d3c3bd3775c765f99ba3789082713eeb34da626ed34"}
Oct 30 19:52:41.402589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.402556    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"]
Oct 30 19:52:41.545993 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.545951    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"]
Oct 30 19:52:41.559853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.558928    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.573550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.573118    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"]
Oct 30 19:52:41.620350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.620257    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.620523 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.620381    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.620599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.620580    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgnql\" (UniqueName: \"kubernetes.io/projected/bbe8f850-4a29-45ec-9068-4072c8859aa7-kube-api-access-zgnql\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.722323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.722280    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.722731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.722616    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgnql\" (UniqueName: \"kubernetes.io/projected/bbe8f850-4a29-45ec-9068-4072c8859aa7-kube-api-access-zgnql\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.722848 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.722829    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.723276 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.723250    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.725290 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.725264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.742538 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.742453    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgnql\" (UniqueName: \"kubernetes.io/projected/bbe8f850-4a29-45ec-9068-4072c8859aa7-kube-api-access-zgnql\") pod \"maas-api-revoke-sub-cf0eef37-77knl\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:41.890722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:41.890623    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:42.210864 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:42.210830    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe8f850_4a29_45ec_9068_4072c8859aa7.slice/crio-a2d4c38d19e6495e2cdc4301ede64ff9c6ceb54d92b600da3d9c407854466e55 WatchSource:0}: Error finding container a2d4c38d19e6495e2cdc4301ede64ff9c6ceb54d92b600da3d9c407854466e55: Status 404 returned error can't find the container with id a2d4c38d19e6495e2cdc4301ede64ff9c6ceb54d92b600da3d9c407854466e55
Oct 30 19:52:42.214752 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.214727    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"]
Oct 30 19:52:42.350820 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.350779    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-ffbe09" event={"ID":"d0db1977-1f53-4971-ab43-2adbcb17d80b","Type":"ContainerStarted","Data":"8c4871eee03111cb7694e7a81bc9e8564b0c5dba02b2180d589921dfa8dab5e0"}
Oct 30 19:52:42.353195 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.353147    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784" event={"ID":"161c49f7-8406-4566-9703-7f76c8ed443b","Type":"ContainerStarted","Data":"cbc52c82f31d3dfd1831d645daed18c80cb7ebcf60e02f2c29b54d3ef3869c25"}
Oct 30 19:52:42.353375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.353215    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784" event={"ID":"161c49f7-8406-4566-9703-7f76c8ed443b","Type":"ContainerStarted","Data":"7ad6ae40b199a140eae1461d3b2e73a79ba0d675bc2efeaf2399f96d48f22482"}
Oct 30 19:52:42.355992 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.355957    4519 generic.go:356] "Generic (PLEG): container finished" podID="9f073363-f521-4176-aae3-7b146033048b" containerID="d9dc91dc8dc01e9fa0d91a26dec4e2cad736f07322dadf40af9cad8a2a91fbca" exitCode=0
Oct 30 19:52:42.356151 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.356052    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv" event={"ID":"9f073363-f521-4176-aae3-7b146033048b","Type":"ContainerDied","Data":"d9dc91dc8dc01e9fa0d91a26dec4e2cad736f07322dadf40af9cad8a2a91fbca"}
Oct 30 19:52:42.358228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.358156    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl" event={"ID":"bbe8f850-4a29-45ec-9068-4072c8859aa7","Type":"ContainerStarted","Data":"a2d4c38d19e6495e2cdc4301ede64ff9c6ceb54d92b600da3d9c407854466e55"}
Oct 30 19:52:42.391603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.391531    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7059-ffbe09" podStartSLOduration=1.604937077 podStartE2EDuration="2.391458893s" totalImagesPullingTime="786.521816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:42.388602527 +0000 UTC m=+1086.889306250" watchObservedRunningTime="2026-10-30 19:52:42.391458893 +0000 UTC m=+1086.892162618"
Oct 30 19:52:42.471211 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:42.471147    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784" podStartSLOduration=2.471128673 podStartE2EDuration="2.471128673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:42.466221907 +0000 UTC m=+1086.966925632" watchObservedRunningTime="2026-10-30 19:52:42.471128673 +0000 UTC m=+1086.971832445"
Oct 30 19:52:43.266663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.266617    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7059-ffbe09"]
Oct 30 19:52:43.275870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.275639    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7059-ffbe09"]
Oct 30 19:52:43.370770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.370735    4519 generic.go:356] "Generic (PLEG): container finished" podID="161c49f7-8406-4566-9703-7f76c8ed443b" containerID="cbc52c82f31d3dfd1831d645daed18c80cb7ebcf60e02f2c29b54d3ef3869c25" exitCode=0
Oct 30 19:52:43.370970 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.370829    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784" event={"ID":"161c49f7-8406-4566-9703-7f76c8ed443b","Type":"ContainerDied","Data":"cbc52c82f31d3dfd1831d645daed18c80cb7ebcf60e02f2c29b54d3ef3869c25"}
Oct 30 19:52:43.373140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.373107    4519 generic.go:356] "Generic (PLEG): container finished" podID="bbe8f850-4a29-45ec-9068-4072c8859aa7" containerID="6942d6709587b4f75dad4e823c5ec093c82fdfd8387cd57995b671d8fdb8c834" exitCode=0
Oct 30 19:52:43.373304 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.373204    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl" event={"ID":"bbe8f850-4a29-45ec-9068-4072c8859aa7","Type":"ContainerDied","Data":"6942d6709587b4f75dad4e823c5ec093c82fdfd8387cd57995b671d8fdb8c834"}
Oct 30 19:52:43.436405 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.436316    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7059-5d48a3"]
Oct 30 19:52:43.437307 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.437282    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d0db1977-1f53-4971-ab43-2adbcb17d80b" containerName="test-curl-7059-ffbe09"
Oct 30 19:52:43.437417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.437407    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0db1977-1f53-4971-ab43-2adbcb17d80b" containerName="test-curl-7059-ffbe09"
Oct 30 19:52:43.437686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.437673    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0db1977-1f53-4971-ab43-2adbcb17d80b" containerName="test-curl-7059-ffbe09"
Oct 30 19:52:43.441711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.441682    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:43.493325 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.493214    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-5d48a3"]
Oct 30 19:52:43.555066 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.555016    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zk2x\" (UniqueName: \"kubernetes.io/projected/9e80f6d4-26f9-4c1f-83c6-31ad37fef405-kube-api-access-9zk2x\") pod \"test-curl-7059-5d48a3\" (UID: \"9e80f6d4-26f9-4c1f-83c6-31ad37fef405\") " pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:43.601751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.601718    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:43.656761 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.656722    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f073363-f521-4176-aae3-7b146033048b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f073363-f521-4176-aae3-7b146033048b-maas-api-openshift-ca\") pod \"9f073363-f521-4176-aae3-7b146033048b\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") "
Oct 30 19:52:43.656950 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.656855    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f073363-f521-4176-aae3-7b146033048b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f073363-f521-4176-aae3-7b146033048b-maas-api-xks-ca\") pod \"9f073363-f521-4176-aae3-7b146033048b\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") "
Oct 30 19:52:43.657072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.657050    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f073363-f521-4176-aae3-7b146033048b-kube-api-access-d4bxz\" (UniqueName: \"kubernetes.io/projected/9f073363-f521-4176-aae3-7b146033048b-kube-api-access-d4bxz\") pod \"9f073363-f521-4176-aae3-7b146033048b\" (UID: \"9f073363-f521-4176-aae3-7b146033048b\") "
Oct 30 19:52:43.657847 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.657816    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zk2x\" (UniqueName: \"kubernetes.io/projected/9e80f6d4-26f9-4c1f-83c6-31ad37fef405-kube-api-access-9zk2x\") pod \"test-curl-7059-5d48a3\" (UID: \"9e80f6d4-26f9-4c1f-83c6-31ad37fef405\") " pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:43.658642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.658613    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f073363-f521-4176-aae3-7b146033048b-maas-api-openshift-ca" pod "9f073363-f521-4176-aae3-7b146033048b" (UID: "9f073363-f521-4176-aae3-7b146033048b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:43.661506 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.661430    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f073363-f521-4176-aae3-7b146033048b-kube-api-access-d4bxz" pod "9f073363-f521-4176-aae3-7b146033048b" (UID: "9f073363-f521-4176-aae3-7b146033048b"). InnerVolumeSpecName "kube-api-access-d4bxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:43.661891 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.661861    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f073363-f521-4176-aae3-7b146033048b-maas-api-xks-ca" pod "9f073363-f521-4176-aae3-7b146033048b" (UID: "9f073363-f521-4176-aae3-7b146033048b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:43.679319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.679280    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zk2x\" (UniqueName: \"kubernetes.io/projected/9e80f6d4-26f9-4c1f-83c6-31ad37fef405-kube-api-access-9zk2x\") pod \"test-curl-7059-5d48a3\" (UID: \"9e80f6d4-26f9-4c1f-83c6-31ad37fef405\") " pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:43.759410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.759253    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f073363-f521-4176-aae3-7b146033048b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:43.759410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.759288    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f073363-f521-4176-aae3-7b146033048b-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:43.759410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.759298    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4bxz\" (UniqueName: \"kubernetes.io/projected/9f073363-f521-4176-aae3-7b146033048b-kube-api-access-d4bxz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:43.794027 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:43.793989    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:44.124606 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:44.124576    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e80f6d4_26f9_4c1f_83c6_31ad37fef405.slice/crio-ea9a0874b7a873046725904221edec53ed5b7eb6f618932774c535d0273d49be WatchSource:0}: Error finding container ea9a0874b7a873046725904221edec53ed5b7eb6f618932774c535d0273d49be: Status 404 returned error can't find the container with id ea9a0874b7a873046725904221edec53ed5b7eb6f618932774c535d0273d49be
Oct 30 19:52:44.127882 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.127853    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-5d48a3"]
Oct 30 19:52:44.390200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.390100    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-5d48a3" event={"ID":"9e80f6d4-26f9-4c1f-83c6-31ad37fef405","Type":"ContainerStarted","Data":"3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58"}
Oct 30 19:52:44.390200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.390154    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-5d48a3" event={"ID":"9e80f6d4-26f9-4c1f-83c6-31ad37fef405","Type":"ContainerStarted","Data":"ea9a0874b7a873046725904221edec53ed5b7eb6f618932774c535d0273d49be"}
Oct 30 19:52:44.392009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.391987    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"
Oct 30 19:52:44.392128 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.392024    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv" event={"ID":"9f073363-f521-4176-aae3-7b146033048b","Type":"ContainerDied","Data":"0eaa96604168a41c61228d3c3bd3775c765f99ba3789082713eeb34da626ed34"}
Oct 30 19:52:44.392128 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.392074    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eaa96604168a41c61228d3c3bd3775c765f99ba3789082713eeb34da626ed34"
Oct 30 19:52:44.392286 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.392266    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7059-ffbe09" podUID="d0db1977-1f53-4971-ab43-2adbcb17d80b" containerName="test-curl-7059-ffbe09" containerID="cri-o://8c4871eee03111cb7694e7a81bc9e8564b0c5dba02b2180d589921dfa8dab5e0" gracePeriod=2
Oct 30 19:52:44.429989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.428424    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7059-5d48a3" podStartSLOduration=1.428410024 podStartE2EDuration="1.428410024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:44.427876844 +0000 UTC m=+1088.928580570" watchObservedRunningTime="2026-10-30 19:52:44.428410024 +0000 UTC m=+1088.929113747"
Oct 30 19:52:44.608732 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.608699    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:44.678197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.678154    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-xks-ca\") pod \"bbe8f850-4a29-45ec-9068-4072c8859aa7\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") "
Oct 30 19:52:44.678453 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.678384    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-openshift-ca\") pod \"bbe8f850-4a29-45ec-9068-4072c8859aa7\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") "
Oct 30 19:52:44.678601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.678583    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe8f850-4a29-45ec-9068-4072c8859aa7-kube-api-access-zgnql\" (UniqueName: \"kubernetes.io/projected/bbe8f850-4a29-45ec-9068-4072c8859aa7-kube-api-access-zgnql\") pod \"bbe8f850-4a29-45ec-9068-4072c8859aa7\" (UID: \"bbe8f850-4a29-45ec-9068-4072c8859aa7\") "
Oct 30 19:52:44.680429 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.680383    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-openshift-ca" pod "bbe8f850-4a29-45ec-9068-4072c8859aa7" (UID: "bbe8f850-4a29-45ec-9068-4072c8859aa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:44.680690 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.680662    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:44.685321 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.685272    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-xks-ca" pod "bbe8f850-4a29-45ec-9068-4072c8859aa7" (UID: "bbe8f850-4a29-45ec-9068-4072c8859aa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:44.686845 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.686806    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe8f850-4a29-45ec-9068-4072c8859aa7-kube-api-access-zgnql" pod "bbe8f850-4a29-45ec-9068-4072c8859aa7" (UID: "bbe8f850-4a29-45ec-9068-4072c8859aa7"). InnerVolumeSpecName "kube-api-access-zgnql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:44.780972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.780902    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161c49f7-8406-4566-9703-7f76c8ed443b-kube-api-access-8mhjc\" (UniqueName: \"kubernetes.io/projected/161c49f7-8406-4566-9703-7f76c8ed443b-kube-api-access-8mhjc\") pod \"161c49f7-8406-4566-9703-7f76c8ed443b\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") "
Oct 30 19:52:44.782306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.781696    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-openshift-ca\") pod \"161c49f7-8406-4566-9703-7f76c8ed443b\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") "
Oct 30 19:52:44.782603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.782581    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-xks-ca\") pod \"161c49f7-8406-4566-9703-7f76c8ed443b\" (UID: \"161c49f7-8406-4566-9703-7f76c8ed443b\") "
Oct 30 19:52:44.783438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.782275    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-openshift-ca" pod "161c49f7-8406-4566-9703-7f76c8ed443b" (UID: "161c49f7-8406-4566-9703-7f76c8ed443b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:44.785025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.784616    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161c49f7-8406-4566-9703-7f76c8ed443b-kube-api-access-8mhjc" pod "161c49f7-8406-4566-9703-7f76c8ed443b" (UID: "161c49f7-8406-4566-9703-7f76c8ed443b"). InnerVolumeSpecName "kube-api-access-8mhjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:44.785025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.784928    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:44.785025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.784951    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:44.785025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.784967    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgnql\" (UniqueName: \"kubernetes.io/projected/bbe8f850-4a29-45ec-9068-4072c8859aa7-kube-api-access-zgnql\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:44.785025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.784982    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe8f850-4a29-45ec-9068-4072c8859aa7-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:44.785025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.784996    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mhjc\" (UniqueName: \"kubernetes.io/projected/161c49f7-8406-4566-9703-7f76c8ed443b-kube-api-access-8mhjc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:44.785978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.785948    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-xks-ca" pod "161c49f7-8406-4566-9703-7f76c8ed443b" (UID: "161c49f7-8406-4566-9703-7f76c8ed443b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:44.886710 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:44.886670    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161c49f7-8406-4566-9703-7f76c8ed443b-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:45.368746 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.368695    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7059-5d48a3"]
Oct 30 19:52:45.378368 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.378328    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7059-5d48a3"]
Oct 30 19:52:45.405831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.405800    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"
Oct 30 19:52:45.406415 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.405815    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl" event={"ID":"bbe8f850-4a29-45ec-9068-4072c8859aa7","Type":"ContainerDied","Data":"a2d4c38d19e6495e2cdc4301ede64ff9c6ceb54d92b600da3d9c407854466e55"}
Oct 30 19:52:45.406519 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.406445    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d4c38d19e6495e2cdc4301ede64ff9c6ceb54d92b600da3d9c407854466e55"
Oct 30 19:52:45.413419 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.413396    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"
Oct 30 19:52:45.413676 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.413423    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784" event={"ID":"161c49f7-8406-4566-9703-7f76c8ed443b","Type":"ContainerDied","Data":"7ad6ae40b199a140eae1461d3b2e73a79ba0d675bc2efeaf2399f96d48f22482"}
Oct 30 19:52:45.413798 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:45.413753    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad6ae40b199a140eae1461d3b2e73a79ba0d675bc2efeaf2399f96d48f22482"
Oct 30 19:52:46.427674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.427637    4519 generic.go:356] "Generic (PLEG): container finished" podID="d0db1977-1f53-4971-ab43-2adbcb17d80b" containerID="8c4871eee03111cb7694e7a81bc9e8564b0c5dba02b2180d589921dfa8dab5e0" exitCode=137
Oct 30 19:52:46.428085 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.428045    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7059-5d48a3" podUID="9e80f6d4-26f9-4c1f-83c6-31ad37fef405" containerName="test-curl-7059-5d48a3" containerID="cri-o://3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58" gracePeriod=2
Oct 30 19:52:46.605629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.605591    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:46.717274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.717233    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0db1977-1f53-4971-ab43-2adbcb17d80b-kube-api-access-wxd26\" (UniqueName: \"kubernetes.io/projected/d0db1977-1f53-4971-ab43-2adbcb17d80b-kube-api-access-wxd26\") pod \"d0db1977-1f53-4971-ab43-2adbcb17d80b\" (UID: \"d0db1977-1f53-4971-ab43-2adbcb17d80b\") "
Oct 30 19:52:46.743489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.735317    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0db1977-1f53-4971-ab43-2adbcb17d80b-kube-api-access-wxd26" pod "d0db1977-1f53-4971-ab43-2adbcb17d80b" (UID: "d0db1977-1f53-4971-ab43-2adbcb17d80b"). InnerVolumeSpecName "kube-api-access-wxd26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.746371    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"]
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747320    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9f073363-f521-4176-aae3-7b146033048b" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747340    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f073363-f521-4176-aae3-7b146033048b" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747358    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bbe8f850-4a29-45ec-9068-4072c8859aa7" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747367    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe8f850-4a29-45ec-9068-4072c8859aa7" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747400    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9e80f6d4-26f9-4c1f-83c6-31ad37fef405" containerName="test-curl-7059-5d48a3"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747409    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e80f6d4-26f9-4c1f-83c6-31ad37fef405" containerName="test-curl-7059-5d48a3"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747433    4519 cpu_manager.go:414] "containerMap: removing container" podUID="161c49f7-8406-4566-9703-7f76c8ed443b" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747441    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161c49f7-8406-4566-9703-7f76c8ed443b" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747627    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="161c49f7-8406-4566-9703-7f76c8ed443b" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747646    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f073363-f521-4176-aae3-7b146033048b" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747657    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe8f850-4a29-45ec-9068-4072c8859aa7" containerName="revoke-keys"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.747669    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e80f6d4-26f9-4c1f-83c6-31ad37fef405" containerName="test-curl-7059-5d48a3"
Oct 30 19:52:46.753770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.752339    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.765488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.762992    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"]
Oct 30 19:52:46.768532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.767615    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ffc83e66-openshift-default-dockercfg-svtrl\""
Oct 30 19:52:46.822149 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.822102    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-envoy\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.822562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.822537    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-token\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.822843 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.822814    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-socket\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.823225 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.822966    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbst\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-kube-api-access-5vbst\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.823759 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.823714    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-credential-socket\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.823925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.823905    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-data\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.824073 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.824057    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-certs\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.824236 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.824221    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istiod-ca-cert\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.824808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.824788    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-podinfo\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.825106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.825088    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxd26\" (UniqueName: \"kubernetes.io/projected/d0db1977-1f53-4971-ab43-2adbcb17d80b-kube-api-access-wxd26\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.927519    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-envoy\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.927603    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-token\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.927692    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-socket\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.927733    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbst\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-kube-api-access-5vbst\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.927838    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-credential-socket\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.927880    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-data\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.927938    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-certs\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.928011    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istiod-ca-cert\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.930494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.928055    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-podinfo\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.984588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.968590    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-data\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.984588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.968873    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-credential-socket\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.984588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.969400    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-certs\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.984588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.969970    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istiod-ca-cert\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.984588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.970241    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-socket\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.984588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.972007    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbst\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-kube-api-access-5vbst\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.984588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.973299    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-podinfo\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:46.999744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:46.999669    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-token\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:47.011251 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.009849    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-envoy\") pod \"e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") " pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:47.087493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.081932    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:47.438991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.438960    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:47.439434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.439014    4519 scope.go:122] "RemoveContainer" containerID="8c4871eee03111cb7694e7a81bc9e8564b0c5dba02b2180d589921dfa8dab5e0"
Oct 30 19:52:47.462224 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:52:47.462179    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-ffbe09\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="d0db1977-1f53-4971-ab43-2adbcb17d80b" pod="opendatahub/test-curl-7059-ffbe09"
Oct 30 19:52:47.585371 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:47.581013    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc1dc0d_e491_406b_bf4c_10d0634917c5.slice/crio-6f590242c47f13eb41a9c556fb6ad8e76af78cb25dd4a0de9e66da655c00563b WatchSource:0}: Error finding container 6f590242c47f13eb41a9c556fb6ad8e76af78cb25dd4a0de9e66da655c00563b: Status 404 returned error can't find the container with id 6f590242c47f13eb41a9c556fb6ad8e76af78cb25dd4a0de9e66da655c00563b
Oct 30 19:52:47.589849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.588529    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:47.589849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.588708    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:47.589849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.588761    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:47.589849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.589407    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"]
Oct 30 19:52:47.632742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:47.632703    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0db1977-1f53-4971-ab43-2adbcb17d80b" path="/var/lib/kubelet/pods/d0db1977-1f53-4971-ab43-2adbcb17d80b/volumes"
Oct 30 19:52:48.171900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.171857    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"]
Oct 30 19:52:48.176770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.176744    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.197922 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.197880    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"]
Oct 30 19:52:48.200821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.200790    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:48.200991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.200906    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:52:48.259188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.259143    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.259436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.259371    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72j6\" (UniqueName: \"kubernetes.io/projected/757dd504-0e6f-4c43-849e-d73c32288a30-kube-api-access-q72j6\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.259674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.259583    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.361550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.361498    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.361770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.361708    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72j6\" (UniqueName: \"kubernetes.io/projected/757dd504-0e6f-4c43-849e-d73c32288a30-kube-api-access-q72j6\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.361900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.361879    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.362357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.362328    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.365014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.364983    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.385113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.385066    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72j6\" (UniqueName: \"kubernetes.io/projected/757dd504-0e6f-4c43-849e-d73c32288a30-kube-api-access-q72j6\") pod \"maas-api-revoke-sub-59ab4f53-mnvpq\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.402934 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.402894    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"]
Oct 30 19:52:48.408310 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.408277    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.424100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.424021    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:52:48.424100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.424023    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:52:48.424100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.424064    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:52:48.424340 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.424099    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f48d6\""
Oct 30 19:52:48.432114 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.432075    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"]
Oct 30 19:52:48.461318 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.459283    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2" event={"ID":"6dc1dc0d-e491-406b-bf4c-10d0634917c5","Type":"ContainerStarted","Data":"4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4"}
Oct 30 19:52:48.461318 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.459360    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2" event={"ID":"6dc1dc0d-e491-406b-bf4c-10d0634917c5","Type":"ContainerStarted","Data":"6f590242c47f13eb41a9c556fb6ad8e76af78cb25dd4a0de9e66da655c00563b"}
Oct 30 19:52:48.495555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.495513    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:48.509553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.507189    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2" podStartSLOduration=2.507167222 podStartE2EDuration="2.507167222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:48.506150706 +0000 UTC m=+1093.006854431" watchObservedRunningTime="2026-10-30 19:52:48.507167222 +0000 UTC m=+1093.007870946"
Oct 30 19:52:48.566492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.565003    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.566492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.565460    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.566492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.565569    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.566492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.565588    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.566492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.565628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.566492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.565962    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f44w\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-kube-api-access-9f44w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.566492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.566046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd58acd-531d-4589-8a3b-b23214045ab4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.677597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.672854    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.677597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.672913    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.677597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.673065    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.677597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.673091    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.677597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.673135    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.677597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.673175    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f44w\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-kube-api-access-9f44w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.677597 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.673219    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd58acd-531d-4589-8a3b-b23214045ab4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.680602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.680568    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.699854 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.686627    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.699854 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.686983    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.699854 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.689315    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.699854 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.692348    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd58acd-531d-4589-8a3b-b23214045ab4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.748538 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.744712    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.775052 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.774998    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f44w\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-kube-api-access-9f44w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk4j8\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:48.813528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.813490    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:48.986956 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.986863    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e80f6d4-26f9-4c1f-83c6-31ad37fef405-kube-api-access-9zk2x\" (UniqueName: \"kubernetes.io/projected/9e80f6d4-26f9-4c1f-83c6-31ad37fef405-kube-api-access-9zk2x\") pod \"9e80f6d4-26f9-4c1f-83c6-31ad37fef405\" (UID: \"9e80f6d4-26f9-4c1f-83c6-31ad37fef405\") "
Oct 30 19:52:48.989134 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:48.989093    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e80f6d4-26f9-4c1f-83c6-31ad37fef405-kube-api-access-9zk2x" pod "9e80f6d4-26f9-4c1f-83c6-31ad37fef405" (UID: "9e80f6d4-26f9-4c1f-83c6-31ad37fef405"). InnerVolumeSpecName "kube-api-access-9zk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:49.023800 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.023762    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:49.029525 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:49.029441    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757dd504_0e6f_4c43_849e_d73c32288a30.slice/crio-0fc95483c1cdc6315ee117042e810470b3f04d4cd55135219fb360c80eaa845c WatchSource:0}: Error finding container 0fc95483c1cdc6315ee117042e810470b3f04d4cd55135219fb360c80eaa845c: Status 404 returned error can't find the container with id 0fc95483c1cdc6315ee117042e810470b3f04d4cd55135219fb360c80eaa845c
Oct 30 19:52:49.032935 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.032903    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"]
Oct 30 19:52:49.083741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.083653    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:49.093783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.093747    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zk2x\" (UniqueName: \"kubernetes.io/projected/9e80f6d4-26f9-4c1f-83c6-31ad37fef405-kube-api-access-9zk2x\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:49.095182 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.095139    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:49.358505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.358453    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"]
Oct 30 19:52:49.364436 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:49.364399    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd58acd_531d_4589_8a3b_b23214045ab4.slice/crio-7c1915941752652f691f01282fbd19923c85f4e7999c42dc9ad1f7793bc753d4 WatchSource:0}: Error finding container 7c1915941752652f691f01282fbd19923c85f4e7999c42dc9ad1f7793bc753d4: Status 404 returned error can't find the container with id 7c1915941752652f691f01282fbd19923c85f4e7999c42dc9ad1f7793bc753d4
Oct 30 19:52:49.472279 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.472242    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8" event={"ID":"edd58acd-531d-4589-8a3b-b23214045ab4","Type":"ContainerStarted","Data":"7c1915941752652f691f01282fbd19923c85f4e7999c42dc9ad1f7793bc753d4"}
Oct 30 19:52:49.474245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.474210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq" event={"ID":"757dd504-0e6f-4c43-849e-d73c32288a30","Type":"ContainerStarted","Data":"7ead5d6ae5a54989b4046623efd5955cb9c3881bbc5c9e80cc1a77b7d27603ca"}
Oct 30 19:52:49.474405 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.474271    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq" event={"ID":"757dd504-0e6f-4c43-849e-d73c32288a30","Type":"ContainerStarted","Data":"0fc95483c1cdc6315ee117042e810470b3f04d4cd55135219fb360c80eaa845c"}
Oct 30 19:52:49.475669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.475644    4519 generic.go:356] "Generic (PLEG): container finished" podID="9e80f6d4-26f9-4c1f-83c6-31ad37fef405" containerID="3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58" exitCode=137
Oct 30 19:52:49.475792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.475683    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:49.475792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.475739    4519 scope.go:122] "RemoveContainer" containerID="3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58"
Oct 30 19:52:49.476221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.476197    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:49.477351 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.477330    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:52:49.491952 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.491913    4519 scope.go:122] "RemoveContainer" containerID="3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58"
Oct 30 19:52:49.492408 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:52:49.492382    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58\": container with ID starting with 3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58 not found: ID does not exist" containerID="3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58"
Oct 30 19:52:49.492604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.492414    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58"} err="failed to get container status \"3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58\": rpc error: code = NotFound desc = could not find container \"3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58\": container with ID starting with 3a8a65ff78e4ad35cf16f37c55bb671411f64757f6ce1d3c34d4adcb994d0e58 not found: ID does not exist"
Oct 30 19:52:49.521765 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.521702    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq" podStartSLOduration=1.521683729 podStartE2EDuration="1.521683729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:49.516123236 +0000 UTC m=+1094.016826959" watchObservedRunningTime="2026-10-30 19:52:49.521683729 +0000 UTC m=+1094.022387447"
Oct 30 19:52:49.529278 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:52:49.529240    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-5d48a3\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="9e80f6d4-26f9-4c1f-83c6-31ad37fef405" pod="opendatahub/test-curl-7059-5d48a3"
Oct 30 19:52:49.663196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:49.663162    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e80f6d4-26f9-4c1f-83c6-31ad37fef405" path="/var/lib/kubelet/pods/9e80f6d4-26f9-4c1f-83c6-31ad37fef405/volumes"
Oct 30 19:52:50.490542 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:50.490501    4519 generic.go:356] "Generic (PLEG): container finished" podID="757dd504-0e6f-4c43-849e-d73c32288a30" containerID="7ead5d6ae5a54989b4046623efd5955cb9c3881bbc5c9e80cc1a77b7d27603ca" exitCode=0
Oct 30 19:52:50.491111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:50.491087    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq" event={"ID":"757dd504-0e6f-4c43-849e-d73c32288a30","Type":"ContainerDied","Data":"7ead5d6ae5a54989b4046623efd5955cb9c3881bbc5c9e80cc1a77b7d27603ca"}
Oct 30 19:52:51.067952 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.067907    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"]
Oct 30 19:52:51.073949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.073916    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.098535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.097108    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"]
Oct 30 19:52:51.232357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.232318    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.232555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.232379    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9r9\" (UniqueName: \"kubernetes.io/projected/4d1600c0-ab78-4328-9ddf-35d06f693e77-kube-api-access-nk9r9\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.232555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.232550    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.334592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.334501    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.334592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.334564    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9r9\" (UniqueName: \"kubernetes.io/projected/4d1600c0-ab78-4328-9ddf-35d06f693e77-kube-api-access-nk9r9\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.335026 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.334996    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.335849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.335820    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.337130 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.337103    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.355350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.355313    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9r9\" (UniqueName: \"kubernetes.io/projected/4d1600c0-ab78-4328-9ddf-35d06f693e77-kube-api-access-nk9r9\") pod \"maas-api-revoke-sub-37a559c5-pqctj\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.405768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.405726    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:51.782870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.782828    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"]
Oct 30 19:52:51.802018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.801671    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:51.804452 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:51.804419    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1600c0_ab78_4328_9ddf_35d06f693e77.slice/crio-625b88f5743dae4f108d0f4c88dfd7557b4eba6c71065eb743bfdb15838870ab WatchSource:0}: Error finding container 625b88f5743dae4f108d0f4c88dfd7557b4eba6c71065eb743bfdb15838870ab: Status 404 returned error can't find the container with id 625b88f5743dae4f108d0f4c88dfd7557b4eba6c71065eb743bfdb15838870ab
Oct 30 19:52:51.952719 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.952685    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-xks-ca\") pod \"757dd504-0e6f-4c43-849e-d73c32288a30\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") "
Oct 30 19:52:51.952875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.952820    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-openshift-ca\") pod \"757dd504-0e6f-4c43-849e-d73c32288a30\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") "
Oct 30 19:52:51.952875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.952858    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757dd504-0e6f-4c43-849e-d73c32288a30-kube-api-access-q72j6\" (UniqueName: \"kubernetes.io/projected/757dd504-0e6f-4c43-849e-d73c32288a30-kube-api-access-q72j6\") pod \"757dd504-0e6f-4c43-849e-d73c32288a30\" (UID: \"757dd504-0e6f-4c43-849e-d73c32288a30\") "
Oct 30 19:52:51.954111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.954082    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-openshift-ca" pod "757dd504-0e6f-4c43-849e-d73c32288a30" (UID: "757dd504-0e6f-4c43-849e-d73c32288a30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:51.956022 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.955988    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757dd504-0e6f-4c43-849e-d73c32288a30-kube-api-access-q72j6" pod "757dd504-0e6f-4c43-849e-d73c32288a30" (UID: "757dd504-0e6f-4c43-849e-d73c32288a30"). InnerVolumeSpecName "kube-api-access-q72j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:51.956409 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:51.956367    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-xks-ca" pod "757dd504-0e6f-4c43-849e-d73c32288a30" (UID: "757dd504-0e6f-4c43-849e-d73c32288a30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:52.055591 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.055554    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:52.055591 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.055588    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757dd504-0e6f-4c43-849e-d73c32288a30-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:52.055591 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.055598    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72j6\" (UniqueName: \"kubernetes.io/projected/757dd504-0e6f-4c43-849e-d73c32288a30-kube-api-access-q72j6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:52.486326 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.486285    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:52.486448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.486359    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:52:52.527852 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.527764    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"
Oct 30 19:52:52.527852 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.527773    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq" event={"ID":"757dd504-0e6f-4c43-849e-d73c32288a30","Type":"ContainerDied","Data":"0fc95483c1cdc6315ee117042e810470b3f04d4cd55135219fb360c80eaa845c"}
Oct 30 19:52:52.528088 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.527861    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc95483c1cdc6315ee117042e810470b3f04d4cd55135219fb360c80eaa845c"
Oct 30 19:52:52.529821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.529620    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj" event={"ID":"4d1600c0-ab78-4328-9ddf-35d06f693e77","Type":"ContainerStarted","Data":"cbfff8ffb3366f4977f6cb4871ddae598329e0d143e97533ab0b9f47b4a27c04"}
Oct 30 19:52:52.530001 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.529850    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj" event={"ID":"4d1600c0-ab78-4328-9ddf-35d06f693e77","Type":"ContainerStarted","Data":"625b88f5743dae4f108d0f4c88dfd7557b4eba6c71065eb743bfdb15838870ab"}
Oct 30 19:52:52.586901 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.586844    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj" podStartSLOduration=1.586826517 podStartE2EDuration="1.586826517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:52.571009181 +0000 UTC m=+1097.071712906" watchObservedRunningTime="2026-10-30 19:52:52.586826517 +0000 UTC m=+1097.087530294"
Oct 30 19:52:53.048041 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:52.986613    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"]
Oct 30 19:52:53.048041 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.040122    4519 cpu_manager.go:414] "containerMap: removing container" podUID="757dd504-0e6f-4c43-849e-d73c32288a30" containerName="revoke-keys"
Oct 30 19:52:53.048041 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.040150    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757dd504-0e6f-4c43-849e-d73c32288a30" containerName="revoke-keys"
Oct 30 19:52:53.048041 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.040427    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="757dd504-0e6f-4c43-849e-d73c32288a30" containerName="revoke-keys"
Oct 30 19:52:53.061572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.061536    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"]
Oct 30 19:52:53.062012 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.061913    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.123760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.123723    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.123953 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.123893    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.124008 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.123992    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhzl\" (UniqueName: \"kubernetes.io/projected/5db124ff-ad32-4eff-93c9-2704963b43b4-kube-api-access-gwhzl\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.225831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.225319    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.225831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.225418    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhzl\" (UniqueName: \"kubernetes.io/projected/5db124ff-ad32-4eff-93c9-2704963b43b4-kube-api-access-gwhzl\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.225831 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.225654    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.226417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.226385    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.232501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.232406    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.248745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.248673    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhzl\" (UniqueName: \"kubernetes.io/projected/5db124ff-ad32-4eff-93c9-2704963b43b4-kube-api-access-gwhzl\") pod \"maas-api-revoke-sub-b447b8c9-cjjvw\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.385246 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.385148    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:53.578508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.577824    4519 generic.go:356] "Generic (PLEG): container finished" podID="4d1600c0-ab78-4328-9ddf-35d06f693e77" containerID="cbfff8ffb3366f4977f6cb4871ddae598329e0d143e97533ab0b9f47b4a27c04" exitCode=0
Oct 30 19:52:53.578508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.578173    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj" event={"ID":"4d1600c0-ab78-4328-9ddf-35d06f693e77","Type":"ContainerDied","Data":"cbfff8ffb3366f4977f6cb4871ddae598329e0d143e97533ab0b9f47b4a27c04"}
Oct 30 19:52:53.584390 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.582689    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8" event={"ID":"edd58acd-531d-4589-8a3b-b23214045ab4","Type":"ContainerStarted","Data":"03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0"}
Oct 30 19:52:53.584390 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.584316    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:53.595017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.594937    4519 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vk4j8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:52:53.595017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.595003    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8" podUID="edd58acd-531d-4589-8a3b-b23214045ab4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:52:53.700696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.700611    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8" podStartSLOduration=2.583055614 podStartE2EDuration="5.700585183s" totalImagesPullingTime="3.117529569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:53.696771778 +0000 UTC m=+1098.197475501" watchObservedRunningTime="2026-10-30 19:52:53.700585183 +0000 UTC m=+1098.201288907"
Oct 30 19:52:53.788526 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:53.784282    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db124ff_ad32_4eff_93c9_2704963b43b4.slice/crio-d3fd83f07b8b3f2a40986b62bfeb17a9add981dff44d1636e4b3e08d5e82bddd WatchSource:0}: Error finding container d3fd83f07b8b3f2a40986b62bfeb17a9add981dff44d1636e4b3e08d5e82bddd: Status 404 returned error can't find the container with id d3fd83f07b8b3f2a40986b62bfeb17a9add981dff44d1636e4b3e08d5e82bddd
Oct 30 19:52:53.788526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:53.786852    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"]
Oct 30 19:52:54.638923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:54.638881    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw" event={"ID":"5db124ff-ad32-4eff-93c9-2704963b43b4","Type":"ContainerStarted","Data":"95784d5b194fa9d954b95c4af63dfa6b9aef51a83e601da834a251e319e0ed41"}
Oct 30 19:52:54.639393 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:54.638973    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw" event={"ID":"5db124ff-ad32-4eff-93c9-2704963b43b4","Type":"ContainerStarted","Data":"d3fd83f07b8b3f2a40986b62bfeb17a9add981dff44d1636e4b3e08d5e82bddd"}
Oct 30 19:52:54.650032 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:54.646195    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 19:52:54.690386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:54.690318    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw" podStartSLOduration=2.690295823 podStartE2EDuration="2.690295823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:54.678792827 +0000 UTC m=+1099.179496564" watchObservedRunningTime="2026-10-30 19:52:54.690295823 +0000 UTC m=+1099.190999548"
Oct 30 19:52:55.004612 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:54.999557    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:55.252660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.252564    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-openshift-ca\") pod \"4d1600c0-ab78-4328-9ddf-35d06f693e77\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") "
Oct 30 19:52:55.252660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.252642    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-xks-ca\") pod \"4d1600c0-ab78-4328-9ddf-35d06f693e77\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") "
Oct 30 19:52:55.257616 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.257568    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-openshift-ca" pod "4d1600c0-ab78-4328-9ddf-35d06f693e77" (UID: "4d1600c0-ab78-4328-9ddf-35d06f693e77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:55.257817 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.257730    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1600c0-ab78-4328-9ddf-35d06f693e77-kube-api-access-nk9r9\" (UniqueName: \"kubernetes.io/projected/4d1600c0-ab78-4328-9ddf-35d06f693e77-kube-api-access-nk9r9\") pod \"4d1600c0-ab78-4328-9ddf-35d06f693e77\" (UID: \"4d1600c0-ab78-4328-9ddf-35d06f693e77\") "
Oct 30 19:52:55.264941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.264192    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:55.393605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.393560    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"]
Oct 30 19:52:55.395652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.395609    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4d1600c0-ab78-4328-9ddf-35d06f693e77" containerName="revoke-keys"
Oct 30 19:52:55.395879 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.395863    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1600c0-ab78-4328-9ddf-35d06f693e77" containerName="revoke-keys"
Oct 30 19:52:55.396402 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.396383    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1600c0-ab78-4328-9ddf-35d06f693e77" containerName="revoke-keys"
Oct 30 19:52:55.407725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.407689    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.415866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.415797    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-xks-ca" pod "4d1600c0-ab78-4328-9ddf-35d06f693e77" (UID: "4d1600c0-ab78-4328-9ddf-35d06f693e77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:55.420992 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.420912    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"]
Oct 30 19:52:55.427062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.426972    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1600c0-ab78-4328-9ddf-35d06f693e77-kube-api-access-nk9r9" pod "4d1600c0-ab78-4328-9ddf-35d06f693e77" (UID: "4d1600c0-ab78-4328-9ddf-35d06f693e77"). InnerVolumeSpecName "kube-api-access-nk9r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:55.485776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.485730    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk9r9\" (UniqueName: \"kubernetes.io/projected/4d1600c0-ab78-4328-9ddf-35d06f693e77-kube-api-access-nk9r9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:55.485776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.485768    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1600c0-ab78-4328-9ddf-35d06f693e77-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:55.587806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.587707    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.587806 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.587754    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlsqs\" (UniqueName: \"kubernetes.io/projected/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-kube-api-access-mlsqs\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.588043 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.587869    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.658702 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.658647    4519 generic.go:356] "Generic (PLEG): container finished" podID="5db124ff-ad32-4eff-93c9-2704963b43b4" containerID="95784d5b194fa9d954b95c4af63dfa6b9aef51a83e601da834a251e319e0ed41" exitCode=0
Oct 30 19:52:55.659210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.658726    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw" event={"ID":"5db124ff-ad32-4eff-93c9-2704963b43b4","Type":"ContainerDied","Data":"95784d5b194fa9d954b95c4af63dfa6b9aef51a83e601da834a251e319e0ed41"}
Oct 30 19:52:55.660707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.660681    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"
Oct 30 19:52:55.660821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.660685    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj" event={"ID":"4d1600c0-ab78-4328-9ddf-35d06f693e77","Type":"ContainerDied","Data":"625b88f5743dae4f108d0f4c88dfd7557b4eba6c71065eb743bfdb15838870ab"}
Oct 30 19:52:55.660821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.660808    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="625b88f5743dae4f108d0f4c88dfd7557b4eba6c71065eb743bfdb15838870ab"
Oct 30 19:52:55.689440 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.689392    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.689691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.689451    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlsqs\" (UniqueName: \"kubernetes.io/projected/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-kube-api-access-mlsqs\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.689691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.689630    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.690297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.690269    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.692622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.692589    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.712398 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.712356    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlsqs\" (UniqueName: \"kubernetes.io/projected/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-kube-api-access-mlsqs\") pod \"maas-api-revoke-sub-3231786e-sbk7n\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:55.738375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:55.738337    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:56.062997 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:52:56.062958    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112f5ed2_38fc_4ac8_9da1_ddaa221551b9.slice/crio-028655e416acebebd26d2d28109f845b46f1f6f7f2b67987b1ae63f06d68de51 WatchSource:0}: Error finding container 028655e416acebebd26d2d28109f845b46f1f6f7f2b67987b1ae63f06d68de51: Status 404 returned error can't find the container with id 028655e416acebebd26d2d28109f845b46f1f6f7f2b67987b1ae63f06d68de51
Oct 30 19:52:56.066875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:56.066821    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"]
Oct 30 19:52:56.715297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:56.715249    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n" event={"ID":"112f5ed2-38fc-4ac8-9da1-ddaa221551b9","Type":"ContainerStarted","Data":"1aa1e4c73cf32d9c7b8b823f10c999be50ca2269cbc29e84784b935ff9bda361"}
Oct 30 19:52:56.715297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:56.715306    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n" event={"ID":"112f5ed2-38fc-4ac8-9da1-ddaa221551b9","Type":"ContainerStarted","Data":"028655e416acebebd26d2d28109f845b46f1f6f7f2b67987b1ae63f06d68de51"}
Oct 30 19:52:56.766880 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:56.765867    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n" podStartSLOduration=1.765840482 podStartE2EDuration="1.765840482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:56.754021397 +0000 UTC m=+1101.254725124" watchObservedRunningTime="2026-10-30 19:52:56.765840482 +0000 UTC m=+1101.266544207"
Oct 30 19:52:56.928081 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:56.928047    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:57.014288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.014179    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-openshift-ca\") pod \"5db124ff-ad32-4eff-93c9-2704963b43b4\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") "
Oct 30 19:52:57.014288 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.014272    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db124ff-ad32-4eff-93c9-2704963b43b4-kube-api-access-gwhzl\" (UniqueName: \"kubernetes.io/projected/5db124ff-ad32-4eff-93c9-2704963b43b4-kube-api-access-gwhzl\") pod \"5db124ff-ad32-4eff-93c9-2704963b43b4\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") "
Oct 30 19:52:57.014566 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.014403    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-xks-ca\") pod \"5db124ff-ad32-4eff-93c9-2704963b43b4\" (UID: \"5db124ff-ad32-4eff-93c9-2704963b43b4\") "
Oct 30 19:52:57.014768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.014737    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-openshift-ca" pod "5db124ff-ad32-4eff-93c9-2704963b43b4" (UID: "5db124ff-ad32-4eff-93c9-2704963b43b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:57.015230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.015208    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:57.017209 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.017178    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-xks-ca" pod "5db124ff-ad32-4eff-93c9-2704963b43b4" (UID: "5db124ff-ad32-4eff-93c9-2704963b43b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:57.017344 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.017320    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db124ff-ad32-4eff-93c9-2704963b43b4-kube-api-access-gwhzl" pod "5db124ff-ad32-4eff-93c9-2704963b43b4" (UID: "5db124ff-ad32-4eff-93c9-2704963b43b4"). InnerVolumeSpecName "kube-api-access-gwhzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:57.117140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.117095    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db124ff-ad32-4eff-93c9-2704963b43b4-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:57.117140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.117134    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhzl\" (UniqueName: \"kubernetes.io/projected/5db124ff-ad32-4eff-93c9-2704963b43b4-kube-api-access-gwhzl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:57.728850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.728806    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw" event={"ID":"5db124ff-ad32-4eff-93c9-2704963b43b4","Type":"ContainerDied","Data":"d3fd83f07b8b3f2a40986b62bfeb17a9add981dff44d1636e4b3e08d5e82bddd"}
Oct 30 19:52:57.729397 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.728881    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3fd83f07b8b3f2a40986b62bfeb17a9add981dff44d1636e4b3e08d5e82bddd"
Oct 30 19:52:57.729397 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.728818    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"
Oct 30 19:52:57.735873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.735833    4519 generic.go:356] "Generic (PLEG): container finished" podID="112f5ed2-38fc-4ac8-9da1-ddaa221551b9" containerID="1aa1e4c73cf32d9c7b8b823f10c999be50ca2269cbc29e84784b935ff9bda361" exitCode=0
Oct 30 19:52:57.736193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:57.736170    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n" event={"ID":"112f5ed2-38fc-4ac8-9da1-ddaa221551b9","Type":"ContainerDied","Data":"1aa1e4c73cf32d9c7b8b823f10c999be50ca2269cbc29e84784b935ff9bda361"}
Oct 30 19:52:58.960245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:58.960205    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:59.047423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.047304    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-openshift-ca\") pod \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") "
Oct 30 19:52:59.047688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.047634    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-kube-api-access-mlsqs\" (UniqueName: \"kubernetes.io/projected/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-kube-api-access-mlsqs\") pod \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") "
Oct 30 19:52:59.047796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.047769    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-openshift-ca" pod "112f5ed2-38fc-4ac8-9da1-ddaa221551b9" (UID: "112f5ed2-38fc-4ac8-9da1-ddaa221551b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:59.048179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.048150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-xks-ca\") pod \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\" (UID: \"112f5ed2-38fc-4ac8-9da1-ddaa221551b9\") "
Oct 30 19:52:59.050307 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.049350    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:59.051696 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.051411    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-xks-ca" pod "112f5ed2-38fc-4ac8-9da1-ddaa221551b9" (UID: "112f5ed2-38fc-4ac8-9da1-ddaa221551b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:59.057095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.057060    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-kube-api-access-mlsqs" pod "112f5ed2-38fc-4ac8-9da1-ddaa221551b9" (UID: "112f5ed2-38fc-4ac8-9da1-ddaa221551b9"). InnerVolumeSpecName "kube-api-access-mlsqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:59.151791 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.151746    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlsqs\" (UniqueName: \"kubernetes.io/projected/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-kube-api-access-mlsqs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:59.151791 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.151791    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112f5ed2-38fc-4ac8-9da1-ddaa221551b9-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:52:59.760520 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.760446    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"
Oct 30 19:52:59.760770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.760520    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n" event={"ID":"112f5ed2-38fc-4ac8-9da1-ddaa221551b9","Type":"ContainerDied","Data":"028655e416acebebd26d2d28109f845b46f1f6f7f2b67987b1ae63f06d68de51"}
Oct 30 19:52:59.760770 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:52:59.760584    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="028655e416acebebd26d2d28109f845b46f1f6f7f2b67987b1ae63f06d68de51"
Oct 30 19:53:02.223956 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:53:02.223896    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:53:02.815622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.815577    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"]
Oct 30 19:53:02.816267 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.816250    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5db124ff-ad32-4eff-93c9-2704963b43b4" containerName="revoke-keys"
Oct 30 19:53:02.816333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.816268    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db124ff-ad32-4eff-93c9-2704963b43b4" containerName="revoke-keys"
Oct 30 19:53:02.816333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.816307    4519 cpu_manager.go:414] "containerMap: removing container" podUID="112f5ed2-38fc-4ac8-9da1-ddaa221551b9" containerName="revoke-keys"
Oct 30 19:53:02.816333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.816313    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112f5ed2-38fc-4ac8-9da1-ddaa221551b9" containerName="revoke-keys"
Oct 30 19:53:02.816431 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.816425    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="112f5ed2-38fc-4ac8-9da1-ddaa221551b9" containerName="revoke-keys"
Oct 30 19:53:02.816498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.816437    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db124ff-ad32-4eff-93c9-2704963b43b4" containerName="revoke-keys"
Oct 30 19:53:02.820851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.820829    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:02.844434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.844395    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"]
Oct 30 19:53:02.850573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.848909    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:02.850573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.849008    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:53:02.933736 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.933694    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:02.933906 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.933874    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gzh\" (UniqueName: \"kubernetes.io/projected/afba94c2-b3ef-4800-91f0-8aa02e2e4785-kube-api-access-79gzh\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:02.933989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:02.933921    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.035580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.035537    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.035792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.035661    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gzh\" (UniqueName: \"kubernetes.io/projected/afba94c2-b3ef-4800-91f0-8aa02e2e4785-kube-api-access-79gzh\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.035792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.035697    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.036381 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.036355    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.038101 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.038079    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.059203 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.059162    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gzh\" (UniqueName: \"kubernetes.io/projected/afba94c2-b3ef-4800-91f0-8aa02e2e4785-kube-api-access-79gzh\") pod \"maas-api-revoke-sub-83440e29-848lz\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.136393 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.136291    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:03.510939 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:53:03.510907    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafba94c2_b3ef_4800_91f0_8aa02e2e4785.slice/crio-d8c1a6c11d12b493df7104ca0dd005809ccc8ede8434207e2b8b4d901c52971b WatchSource:0}: Error finding container d8c1a6c11d12b493df7104ca0dd005809ccc8ede8434207e2b8b4d901c52971b: Status 404 returned error can't find the container with id d8c1a6c11d12b493df7104ca0dd005809ccc8ede8434207e2b8b4d901c52971b
Oct 30 19:53:03.514794 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.514769    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"]
Oct 30 19:53:03.807907 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:03.807851    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz" event={"ID":"afba94c2-b3ef-4800-91f0-8aa02e2e4785","Type":"ContainerStarted","Data":"d8c1a6c11d12b493df7104ca0dd005809ccc8ede8434207e2b8b4d901c52971b"}
Oct 30 19:53:04.822531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:04.822452    4519 generic.go:356] "Generic (PLEG): container finished" podID="afba94c2-b3ef-4800-91f0-8aa02e2e4785" containerID="1c8ca4f232c02c76b22f1c8310e8047322976b3258c9ab93ced01c6c60ff8602" exitCode=0
Oct 30 19:53:04.822531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:04.822501    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz" event={"ID":"afba94c2-b3ef-4800-91f0-8aa02e2e4785","Type":"ContainerDied","Data":"1c8ca4f232c02c76b22f1c8310e8047322976b3258c9ab93ced01c6c60ff8602"}
Oct 30 19:53:06.431618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.409981    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"]
Oct 30 19:53:06.550709 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.540635    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.657493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.649560    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"]
Oct 30 19:53:06.753486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.749038    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8r2\" (UniqueName: \"kubernetes.io/projected/041aad9a-34bc-4633-9fe6-a57d8b662011-kube-api-access-cm8r2\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.753486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.749263    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.753486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.749329    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.857553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.854282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8r2\" (UniqueName: \"kubernetes.io/projected/041aad9a-34bc-4633-9fe6-a57d8b662011-kube-api-access-cm8r2\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.865526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.858098    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.870531 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.862404    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.876055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.872213    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.935497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.932610    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8r2\" (UniqueName: \"kubernetes.io/projected/041aad9a-34bc-4633-9fe6-a57d8b662011-kube-api-access-cm8r2\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:06.992501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:06.990984    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0e74899-g9cqx\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:07.235493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.231587    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:07.235493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.233192    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:07.248508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.237387    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:07.248508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.237814    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:53:07.248508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.238017    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:53:07.248508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.238179    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:53:07.258446 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.258403    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:07.283835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.283782    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:07.379175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.379066    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:53:07.380509 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.379440    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:07.968717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:07.968662    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 19:53:08.058262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.058192    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded"
Oct 30 19:53:08.290559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.270644    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:08.343311 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.291331    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.434083    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.291423    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.291450    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.291492    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.291530    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.326095    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.343819    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.435652    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.435701    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.435740    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.435780    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.435819    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.435856    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:53:08.450495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.426211    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.492498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.291400    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.492498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.453003    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:53:08.492498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.466420    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.492498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.472373    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.492498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:08.472830    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:53:09.537148 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.458511    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:53:09.674402 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.467069    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:53:09.674751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.674705    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:09.674901 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.535182    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:09.675349 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.535140    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:09.677100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.535206    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:53:09.758583 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.535242    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:53:09.759900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.536541    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:09.764647 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.536594    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:53:09.781503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:09.536635    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:53:10.099763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.083530    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:53:10.099763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.083636    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:10.462744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.433529    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:10.521943 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.521891    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:53:10.522628 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.522589    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:10.600535 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.574078    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:10.700540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.698330    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:10.700540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.698817    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:10.700540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.699115    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:10.760834 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.760724    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:10.772494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.764930    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:10.798494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.773389    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:53:10.824495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.814924    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:10.825364 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.825323    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-openshift-ca\") pod \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") "
Oct 30 19:53:10.833497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.829878    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-xks-ca\") pod \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") "
Oct 30 19:53:10.863494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.838142    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afba94c2-b3ef-4800-91f0-8aa02e2e4785-kube-api-access-79gzh\" (UniqueName: \"kubernetes.io/projected/afba94c2-b3ef-4800-91f0-8aa02e2e4785-kube-api-access-79gzh\") pod \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\" (UID: \"afba94c2-b3ef-4800-91f0-8aa02e2e4785\") "
Oct 30 19:53:10.988780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:10.955432    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-openshift-ca" pod "afba94c2-b3ef-4800-91f0-8aa02e2e4785" (UID: "afba94c2-b3ef-4800-91f0-8aa02e2e4785"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:11.017036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.016901    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:11.077540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.067553    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-xks-ca" pod "afba94c2-b3ef-4800-91f0-8aa02e2e4785" (UID: "afba94c2-b3ef-4800-91f0-8aa02e2e4785"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:11.120494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.106632    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afba94c2-b3ef-4800-91f0-8aa02e2e4785-kube-api-access-79gzh" pod "afba94c2-b3ef-4800-91f0-8aa02e2e4785" (UID: "afba94c2-b3ef-4800-91f0-8aa02e2e4785"). InnerVolumeSpecName "kube-api-access-79gzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:11.184230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.184192    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:11.184514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.184495    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gzh\" (UniqueName: \"kubernetes.io/projected/afba94c2-b3ef-4800-91f0-8aa02e2e4785-kube-api-access-79gzh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:11.184669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.184652    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afba94c2-b3ef-4800-91f0-8aa02e2e4785-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:11.279846 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:53:11.279748    4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.636s"
Oct 30 19:53:11.288496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.280261    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:53:11.454490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.443259    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:11.701496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.479284    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx" event={"ID":"041aad9a-34bc-4633-9fe6-a57d8b662011","Type":"ContainerStarted","Data":"fea9d98f8d0a0961bac0704d8bbf7cee720cc4d6f4f170e177befb7adeeb669b"}
Oct 30 19:53:11.714562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.522314    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:11.715054 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.715007    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:53:11.715783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.715730    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:11.717271 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.717237    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:11.757132 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.757099    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"
Oct 30 19:53:11.826525 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.778585    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:53:11.826525 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.778539    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:11.826525 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.798941    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:11.863533 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:11.842680    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:53:12.357639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.357208    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:12.491495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.488867    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz" event={"ID":"afba94c2-b3ef-4800-91f0-8aa02e2e4785","Type":"ContainerDied","Data":"d8c1a6c11d12b493df7104ca0dd005809ccc8ede8434207e2b8b4d901c52971b"}
Oct 30 19:53:12.491495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.488947    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8c1a6c11d12b493df7104ca0dd005809ccc8ede8434207e2b8b4d901c52971b"
Oct 30 19:53:12.491495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.489038    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:53:12.491495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.489061    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:53:12.692285 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.684913    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:53:12.692285 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.684977    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:12.699176 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.699129    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:53:12.731175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.731112    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:53:12.743501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.733729    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:12.743501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.733770    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:53:12.743501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.733790    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:12.805344 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.805225    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:12.828359 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.828309    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:53:12.861592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:12.856808    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:13.089760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.089657    4519 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:53:13.136140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.134797    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:53:13.482489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.481770    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"]
Oct 30 19:53:13.520490 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:53:13.520046    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafba94c2_b3ef_4800_91f0_8aa02e2e4785.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:53:13.734414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.734305    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:13.762493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.736597    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:53:13.768534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.762007    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:13.816490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.762332    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:13.884500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.763832    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:13.893510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.888581    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:53:13.897511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.894676    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:13.904021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:13.845630    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:14.324512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.244550    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:14.788553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.735655    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:53:14.788553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.767578    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:14.805635 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.805553    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:14.892739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.892682    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:14.895141 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.895099    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:14.895542 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.895512    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:53:14.934398 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.934343    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:14.940651 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:14.937934    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:53:15.492986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.492927    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:15.542756 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.542694    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:53:15.754008 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.753872    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:15.812239 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.812191    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:15.846222 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.846181    4519 generic.go:356] "Generic (PLEG): container finished" podID="041aad9a-34bc-4633-9fe6-a57d8b662011" containerID="a04bb3d251bc055230d4fbed62863dfc7de075e941e468ae29251e7463b54de1" exitCode=0
Oct 30 19:53:15.853488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.851595    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:15.925495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.911120    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:15.942792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.919542    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:15.952141 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.943095    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:15.952735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.943666    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:53:15.953255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.946690    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx" event={"ID":"041aad9a-34bc-4633-9fe6-a57d8b662011","Type":"ContainerDied","Data":"a04bb3d251bc055230d4fbed62863dfc7de075e941e468ae29251e7463b54de1"}
Oct 30 19:53:15.958629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:15.958581    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:16.369216 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.369176    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"]
Oct 30 19:53:16.383950 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.383912    4519 cpu_manager.go:414] "containerMap: removing container" podUID="afba94c2-b3ef-4800-91f0-8aa02e2e4785" containerName="revoke-keys"
Oct 30 19:53:16.384206 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.384185    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afba94c2-b3ef-4800-91f0-8aa02e2e4785" containerName="revoke-keys"
Oct 30 19:53:16.384695 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.384672    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="afba94c2-b3ef-4800-91f0-8aa02e2e4785" containerName="revoke-keys"
Oct 30 19:53:16.519493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.497839    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.555170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.551224    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"]
Oct 30 19:53:16.555170 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.551638    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:16.663232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.662675    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ckc\" (UniqueName: \"kubernetes.io/projected/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-kube-api-access-v6ckc\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.663232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.662780    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.663232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.662923    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.761222 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.760624    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:53:16.774537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.770023    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ckc\" (UniqueName: \"kubernetes.io/projected/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-kube-api-access-v6ckc\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.774537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.770147    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.774537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.770290    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.786493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.783729    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.794494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.794025    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.814411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.813854    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:16.821513 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.821461    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ckc\" (UniqueName: \"kubernetes.io/projected/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-kube-api-access-v6ckc\") pod \"maas-api-revoke-sub-3728ca80-2rf82\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.906490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.887848    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:53:16.934521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.934409    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:16.939488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.936403    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:53:16.953513 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.953248    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:16.968923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.966279    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:16.968923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.967235    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:16.968923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:16.967555    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:17.109294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.109255    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:53:17.160516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.123163    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:53:17.168922 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.161233    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:53:17.323487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.302731    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:53:17.323487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.322665    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:53:17.366096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.364459    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:53:17.404116 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.404079    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:53:17.412066 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.412036    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:53:17.441927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.440671    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:53:17.709577 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.709536    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:17.738092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.738056    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-openshift-ca\") pod \"041aad9a-34bc-4633-9fe6-a57d8b662011\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") "
Oct 30 19:53:17.738298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.738195    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-xks-ca\") pod \"041aad9a-34bc-4633-9fe6-a57d8b662011\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") "
Oct 30 19:53:17.738398 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.738380    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041aad9a-34bc-4633-9fe6-a57d8b662011-kube-api-access-cm8r2\" (UniqueName: \"kubernetes.io/projected/041aad9a-34bc-4633-9fe6-a57d8b662011-kube-api-access-cm8r2\") pod \"041aad9a-34bc-4633-9fe6-a57d8b662011\" (UID: \"041aad9a-34bc-4633-9fe6-a57d8b662011\") "
Oct 30 19:53:17.738537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.738506    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-openshift-ca" pod "041aad9a-34bc-4633-9fe6-a57d8b662011" (UID: "041aad9a-34bc-4633-9fe6-a57d8b662011"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:17.739162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.739129    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:17.740625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.740598    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-xks-ca" pod "041aad9a-34bc-4633-9fe6-a57d8b662011" (UID: "041aad9a-34bc-4633-9fe6-a57d8b662011"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:17.741064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.741049    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041aad9a-34bc-4633-9fe6-a57d8b662011-kube-api-access-cm8r2" pod "041aad9a-34bc-4633-9fe6-a57d8b662011" (UID: "041aad9a-34bc-4633-9fe6-a57d8b662011"). InnerVolumeSpecName "kube-api-access-cm8r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:17.788483 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.780751    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"]
Oct 30 19:53:17.795060 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:53:17.794673    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65eeb886_ac6c_42c5_8cb4_00319cb62ae2.slice/crio-d12f447111e6757988e2337c37f680319c0347b41aba16cd1fcfb1f05435c9ee WatchSource:0}: Error finding container d12f447111e6757988e2337c37f680319c0347b41aba16cd1fcfb1f05435c9ee: Status 404 returned error can't find the container with id d12f447111e6757988e2337c37f680319c0347b41aba16cd1fcfb1f05435c9ee
Oct 30 19:53:17.840459 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.840416    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/041aad9a-34bc-4633-9fe6-a57d8b662011-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:17.840459 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:17.840456    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8r2\" (UniqueName: \"kubernetes.io/projected/041aad9a-34bc-4633-9fe6-a57d8b662011-kube-api-access-cm8r2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:18.004159 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.004067    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"
Oct 30 19:53:18.004159 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.004109    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx" event={"ID":"041aad9a-34bc-4633-9fe6-a57d8b662011","Type":"ContainerDied","Data":"fea9d98f8d0a0961bac0704d8bbf7cee720cc4d6f4f170e177befb7adeeb669b"}
Oct 30 19:53:18.004407 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.004174    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fea9d98f8d0a0961bac0704d8bbf7cee720cc4d6f4f170e177befb7adeeb669b"
Oct 30 19:53:18.006032 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.005994    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82" event={"ID":"65eeb886-ac6c-42c5-8cb4-00319cb62ae2","Type":"ContainerStarted","Data":"d12f447111e6757988e2337c37f680319c0347b41aba16cd1fcfb1f05435c9ee"}
Oct 30 19:53:18.895904 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.895854    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"]
Oct 30 19:53:18.896754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.896732    4519 cpu_manager.go:414] "containerMap: removing container" podUID="041aad9a-34bc-4633-9fe6-a57d8b662011" containerName="revoke-keys"
Oct 30 19:53:18.896754 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.896753    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041aad9a-34bc-4633-9fe6-a57d8b662011" containerName="revoke-keys"
Oct 30 19:53:18.896898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.896875    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="041aad9a-34bc-4633-9fe6-a57d8b662011" containerName="revoke-keys"
Oct 30 19:53:18.900960 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.900937    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:18.921442 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.921409    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"]
Oct 30 19:53:18.961751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.961628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:18.962001 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.961967    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhdm\" (UniqueName: \"kubernetes.io/projected/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-kube-api-access-rxhdm\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:18.962279 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:18.962255    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.018210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.018170    4519 generic.go:356] "Generic (PLEG): container finished" podID="65eeb886-ac6c-42c5-8cb4-00319cb62ae2" containerID="5502876e5be5137c8ce734d7dffa951e7895babdd956ad46fb43d5d0d40a736b" exitCode=0
Oct 30 19:53:19.018417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.018260    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82" event={"ID":"65eeb886-ac6c-42c5-8cb4-00319cb62ae2","Type":"ContainerDied","Data":"5502876e5be5137c8ce734d7dffa951e7895babdd956ad46fb43d5d0d40a736b"}
Oct 30 19:53:19.063990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.063895    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.063990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.063966    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.064298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.064124    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhdm\" (UniqueName: \"kubernetes.io/projected/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-kube-api-access-rxhdm\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.065233 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.065202    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.072979 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.072938    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.084898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.084860    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhdm\" (UniqueName: \"kubernetes.io/projected/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-kube-api-access-rxhdm\") pod \"maas-api-revoke-sub-246f440f-qc9xb\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.231885 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.231845    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:19.604398 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:53:19.604305    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c2d0cc_9565_45de_9cb0_f214c8aa6fc8.slice/crio-dba3ba1fad322150acd5573c9c2346101063929e20fbb14b5225f0475dc86a8f WatchSource:0}: Error finding container dba3ba1fad322150acd5573c9c2346101063929e20fbb14b5225f0475dc86a8f: Status 404 returned error can't find the container with id dba3ba1fad322150acd5573c9c2346101063929e20fbb14b5225f0475dc86a8f
Oct 30 19:53:19.625629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:19.625590    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"]
Oct 30 19:53:20.078402 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.078340    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb" event={"ID":"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8","Type":"ContainerStarted","Data":"dba3ba1fad322150acd5573c9c2346101063929e20fbb14b5225f0475dc86a8f"}
Oct 30 19:53:20.336973 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.336939    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:20.414683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.414653    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-kube-api-access-v6ckc\" (UniqueName: \"kubernetes.io/projected/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-kube-api-access-v6ckc\") pod \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") "
Oct 30 19:53:20.414909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.414863    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-openshift-ca\") pod \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") "
Oct 30 19:53:20.415142 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.415076    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-xks-ca\") pod \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\" (UID: \"65eeb886-ac6c-42c5-8cb4-00319cb62ae2\") "
Oct 30 19:53:20.415352 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.415315    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-openshift-ca" pod "65eeb886-ac6c-42c5-8cb4-00319cb62ae2" (UID: "65eeb886-ac6c-42c5-8cb4-00319cb62ae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:20.416107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.416066    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:20.417433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.417407    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-kube-api-access-v6ckc" pod "65eeb886-ac6c-42c5-8cb4-00319cb62ae2" (UID: "65eeb886-ac6c-42c5-8cb4-00319cb62ae2"). InnerVolumeSpecName "kube-api-access-v6ckc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:20.417858 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.417830    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-xks-ca" pod "65eeb886-ac6c-42c5-8cb4-00319cb62ae2" (UID: "65eeb886-ac6c-42c5-8cb4-00319cb62ae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:20.517720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.517675    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ckc\" (UniqueName: \"kubernetes.io/projected/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-kube-api-access-v6ckc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:20.517720 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:20.517712    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65eeb886-ac6c-42c5-8cb4-00319cb62ae2-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:21.094544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:21.094500    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb" event={"ID":"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8","Type":"ContainerDied","Data":"1ba42a64acd05a7170ba3c377567cf2f8cfc0d5c1556d921b8dc9a1cd0eefe29"}
Oct 30 19:53:21.095007 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:21.094507    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" containerID="1ba42a64acd05a7170ba3c377567cf2f8cfc0d5c1556d921b8dc9a1cd0eefe29" exitCode=0
Oct 30 19:53:21.096904 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:21.096865    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82" event={"ID":"65eeb886-ac6c-42c5-8cb4-00319cb62ae2","Type":"ContainerDied","Data":"d12f447111e6757988e2337c37f680319c0347b41aba16cd1fcfb1f05435c9ee"}
Oct 30 19:53:21.097073 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:21.096939    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d12f447111e6757988e2337c37f680319c0347b41aba16cd1fcfb1f05435c9ee"
Oct 30 19:53:21.097073 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:21.096887    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"
Oct 30 19:53:22.297106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.297068    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:22.355992 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.355949    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-kube-api-access-rxhdm\" (UniqueName: \"kubernetes.io/projected/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-kube-api-access-rxhdm\") pod \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") "
Oct 30 19:53:22.356232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.356073    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-xks-ca\") pod \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") "
Oct 30 19:53:22.356232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.356184    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-openshift-ca\") pod \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\" (UID: \"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8\") "
Oct 30 19:53:22.357348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.357313    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-openshift-ca" pod "d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" (UID: "d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:22.379027 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.378978    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-kube-api-access-rxhdm" pod "d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" (UID: "d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8"). InnerVolumeSpecName "kube-api-access-rxhdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:22.380718 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.380664    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-xks-ca" pod "d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" (UID: "d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:22.459368 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.459303    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxhdm\" (UniqueName: \"kubernetes.io/projected/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-kube-api-access-rxhdm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:22.459368 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.459348    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:22.459368 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:22.459365    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:23.120620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:23.120590    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"
Oct 30 19:53:23.120788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:23.120590    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb" event={"ID":"d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8","Type":"ContainerDied","Data":"dba3ba1fad322150acd5573c9c2346101063929e20fbb14b5225f0475dc86a8f"}
Oct 30 19:53:23.120788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:23.120726    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dba3ba1fad322150acd5573c9c2346101063929e20fbb14b5225f0475dc86a8f"
Oct 30 19:53:28.041378 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:28.041335    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"]
Oct 30 19:53:28.059790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:28.059746    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8950b1-dgkxp"]
Oct 30 19:53:29.689490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:29.668247    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6451a9a8-6593-4f80-be6f-3d6b082fafab" path="/var/lib/kubelet/pods/6451a9a8-6593-4f80-be6f-3d6b082fafab/volumes"
Oct 30 19:53:32.037150 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:32.037107    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"]
Oct 30 19:53:32.048024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:32.047981    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3130ac33-c7q42"]
Oct 30 19:53:32.225021 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:53:32.224976    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:53:33.618748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:33.618708    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eca6708-9134-4cde-ad50-7129cac56ea4" path="/var/lib/kubelet/pods/5eca6708-9134-4cde-ad50-7129cac56ea4/volumes"
Oct 30 19:53:36.793512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.792848    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"]
Oct 30 19:53:36.794031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.793915    4519 cpu_manager.go:414] "containerMap: removing container" podUID="65eeb886-ac6c-42c5-8cb4-00319cb62ae2" containerName="revoke-keys"
Oct 30 19:53:36.794031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.793937    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65eeb886-ac6c-42c5-8cb4-00319cb62ae2" containerName="revoke-keys"
Oct 30 19:53:36.794031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.793980    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" containerName="revoke-keys"
Oct 30 19:53:36.794031 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.793989    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" containerName="revoke-keys"
Oct 30 19:53:36.794234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.794144    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="65eeb886-ac6c-42c5-8cb4-00319cb62ae2" containerName="revoke-keys"
Oct 30 19:53:36.794234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.794159    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" containerName="revoke-keys"
Oct 30 19:53:36.800602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.800530    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.807790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.806920    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"]
Oct 30 19:53:36.832745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.828656    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:36.832745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.828699    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:53:36.845245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.845184    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.845538 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.845371    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.845538 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.845431    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqk2\" (UniqueName: \"kubernetes.io/projected/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-kube-api-access-rmqk2\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.947279 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.947238    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.947480 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.947372    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.947480 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.947432    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqk2\" (UniqueName: \"kubernetes.io/projected/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-kube-api-access-rmqk2\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.948244 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.948215    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.949891 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.949866    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:36.969992 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:36.968851    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqk2\" (UniqueName: \"kubernetes.io/projected/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-kube-api-access-rmqk2\") pod \"maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:37.129639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:37.129558    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:37.468663 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:53:37.468122    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53d8e9e_82a2_4154_a4fd_d0d7546bcf3e.slice/crio-9054cfcb211f98722aa0152787b17d5f63175a20cdd94352e0e9adbca27ca200 WatchSource:0}: Error finding container 9054cfcb211f98722aa0152787b17d5f63175a20cdd94352e0e9adbca27ca200: Status 404 returned error can't find the container with id 9054cfcb211f98722aa0152787b17d5f63175a20cdd94352e0e9adbca27ca200
Oct 30 19:53:37.472765 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:37.472735    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"]
Oct 30 19:53:38.363432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:38.363382    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr" event={"ID":"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e","Type":"ContainerStarted","Data":"b419a095d132e6a2b256b70169e09481e0119531a3712c81517ea8a72ebfdfc0"}
Oct 30 19:53:38.363981 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:38.363456    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr" event={"ID":"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e","Type":"ContainerStarted","Data":"9054cfcb211f98722aa0152787b17d5f63175a20cdd94352e0e9adbca27ca200"}
Oct 30 19:53:38.416115 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:38.416044    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr" podStartSLOduration=2.416023777 podStartE2EDuration="2.416023777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:38.409047608 +0000 UTC m=+1142.909751333" watchObservedRunningTime="2026-10-30 19:53:38.416023777 +0000 UTC m=+1142.916727550"
Oct 30 19:53:39.378582 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:39.378539    4519 generic.go:356] "Generic (PLEG): container finished" podID="f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" containerID="b419a095d132e6a2b256b70169e09481e0119531a3712c81517ea8a72ebfdfc0" exitCode=0
Oct 30 19:53:39.379038 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:39.378581    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr" event={"ID":"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e","Type":"ContainerDied","Data":"b419a095d132e6a2b256b70169e09481e0119531a3712c81517ea8a72ebfdfc0"}
Oct 30 19:53:40.035872 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.035836    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"]
Oct 30 19:53:40.047983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.047943    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec88c6f-74xjp"]
Oct 30 19:53:40.595425 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.595389    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:40.715512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.715450    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-xks-ca\") pod \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") "
Oct 30 19:53:40.715741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.715634    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-openshift-ca\") pod \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") "
Oct 30 19:53:40.715815 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.715797    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-kube-api-access-rmqk2\" (UniqueName: \"kubernetes.io/projected/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-kube-api-access-rmqk2\") pod \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\" (UID: \"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e\") "
Oct 30 19:53:40.716063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.716035    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-openshift-ca" pod "f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" (UID: "f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:40.716875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.716854    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:40.718626 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.718595    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-xks-ca" pod "f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" (UID: "f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:40.718772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.718630    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-kube-api-access-rmqk2" pod "f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" (UID: "f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e"). InnerVolumeSpecName "kube-api-access-rmqk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:40.818665 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.818621    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqk2\" (UniqueName: \"kubernetes.io/projected/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-kube-api-access-rmqk2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:40.818665 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:40.818665    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:41.405525 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:41.405494    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"
Oct 30 19:53:41.405724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:41.405497    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr" event={"ID":"f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e","Type":"ContainerDied","Data":"9054cfcb211f98722aa0152787b17d5f63175a20cdd94352e0e9adbca27ca200"}
Oct 30 19:53:41.405724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:41.405628    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9054cfcb211f98722aa0152787b17d5f63175a20cdd94352e0e9adbca27ca200"
Oct 30 19:53:41.614376 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:41.614338    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2b8778-84f0-4bf7-b2d2-c638cd709614" path="/var/lib/kubelet/pods/cf2b8778-84f0-4bf7-b2d2-c638cd709614/volumes"
Oct 30 19:53:46.740642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.740573    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:46.799500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.777569    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:46.838496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.777632    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:46.846500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.777660    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:46.870729 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.870671    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:46.874492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.786563    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:53:46.942493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.832906    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:46.961967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.915583    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:46.984982 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:46.948538    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.031496 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.030611    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.058607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.051762    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:48.058607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.053139    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:48.058607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.053504    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.058607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.053881    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.058607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.054092    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.087789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.071729    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.087789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.073102    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.087789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.073452    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:53:48.109562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.100092    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:48.183500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.181783    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded"
Oct 30 19:53:48.286564 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.281026    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:48.445858 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.441731    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 19:53:48.459018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.458967    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:48.554860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.554757    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:53:48.585762 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.585727    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:53:48.586107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.586089    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:53:48.586284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.586270    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:53:48.586443 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.586431    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:53:48.586649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.586634    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:53:48.586813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.586800    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:53:48.586965 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.586952    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:53:48.587108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:48.587095    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:53:49.027491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.023200    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.052688    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.080839    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.094258    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.095567    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.116143    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.122642    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.134840    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:53:49.215955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:49.135173    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:50.068494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.053997    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.068494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.054487    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.089579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.084248    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.099492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.096297    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.106795 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.106741    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:53:50.133873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.118886    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:53:50.167700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.135282    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.167700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.135508    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:53:50.167700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.135676    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.393587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.388817    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.666817 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.666764    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:53:50.667062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.666864    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:50.942553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:50.937578    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:51.072513 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.066151    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:51.099500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.093708    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:51.129493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.101163    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:51.140175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.135670    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:51.140175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.135794    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:53:51.140175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.127704    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:51.140175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.136656    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:53:51.235549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:51.127637    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.067727    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.084328    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.084432    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.084453    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"]
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.084539    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.084566    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de6122-wk8fm"]
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.084592    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"]
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.086143    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" containerName="revoke-keys"
Oct 30 19:53:52.089526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.086170    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" containerName="revoke-keys"
Oct 30 19:53:52.136280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.136230    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:52.138380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.138335    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:52.143731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.139860    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:53:52.173002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.168643    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" containerName="revoke-keys"
Oct 30 19:53:52.211334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.211158    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"]
Oct 30 19:53:52.211708 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.211685    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.222123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.222076    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:53:52.251261 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.251207    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgd8p\" (UniqueName: \"kubernetes.io/projected/40286f5c-ba40-4ea8-8c7e-c36094d15bec-kube-api-access-lgd8p\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.251512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.251388    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.251622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.251520    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.265095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.265053    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:53:52.265412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.265392    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:52.308579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.308546    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 19:53:52.359744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.357991    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.360092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.360055    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.360688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.360658    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgd8p\" (UniqueName: \"kubernetes.io/projected/40286f5c-ba40-4ea8-8c7e-c36094d15bec-kube-api-access-lgd8p\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.362928 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.362882    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.627582 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.625657    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 19:53:52.631071 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.630658    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.638403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.638353    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgd8p\" (UniqueName: \"kubernetes.io/projected/40286f5c-ba40-4ea8-8c7e-c36094d15bec-kube-api-access-lgd8p\") pod \"maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:52.641334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.641298    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 19:53:52.678877 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.678840    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 19:53:52.720228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.720186    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 19:53:52.925010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:52.924966    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:53.131055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:53.130976    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"]
Oct 30 19:53:53.161082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:53.161025    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cacd41-ltbvf"]
Oct 30 19:53:53.589974 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:53.589884    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"]
Oct 30 19:53:53.619023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:53.618979    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d317c0-ef23-46f6-908e-9005eef1dbe9" path="/var/lib/kubelet/pods/66d317c0-ef23-46f6-908e-9005eef1dbe9/volumes"
Oct 30 19:53:53.619724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:53.619704    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ea48e4-a64f-484d-acec-e5b5272b0801" path="/var/lib/kubelet/pods/a9ea48e4-a64f-484d-acec-e5b5272b0801/volumes"
Oct 30 19:53:54.417799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:54.417762    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f" event={"ID":"40286f5c-ba40-4ea8-8c7e-c36094d15bec","Type":"ContainerStarted","Data":"b137dfd03ab4aebdeae57ba17d8ad1f96aff974fb063cbf2c7a166e8fbe033c6"}
Oct 30 19:53:54.418252 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:54.417826    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f" event={"ID":"40286f5c-ba40-4ea8-8c7e-c36094d15bec","Type":"ContainerStarted","Data":"f0c987eeecbb71a52b7686d4da98b590359e344a97d86ea52bc6f0e9d6e06735"}
Oct 30 19:53:54.501459 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:54.499781    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f" podStartSLOduration=4.499759436 podStartE2EDuration="4.499759436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:54.466646423 +0000 UTC m=+1158.967350147" watchObservedRunningTime="2026-10-30 19:53:54.499759436 +0000 UTC m=+1159.000463161"
Oct 30 19:53:55.429238 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:55.429196    4519 generic.go:356] "Generic (PLEG): container finished" podID="40286f5c-ba40-4ea8-8c7e-c36094d15bec" containerID="b137dfd03ab4aebdeae57ba17d8ad1f96aff974fb063cbf2c7a166e8fbe033c6" exitCode=0
Oct 30 19:53:55.429667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:55.429277    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f" event={"ID":"40286f5c-ba40-4ea8-8c7e-c36094d15bec","Type":"ContainerDied","Data":"b137dfd03ab4aebdeae57ba17d8ad1f96aff974fb063cbf2c7a166e8fbe033c6"}
Oct 30 19:53:56.672312 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.672274    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:53:56.811812 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.811712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-xks-ca\") pod \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") "
Oct 30 19:53:56.812009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.811827    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40286f5c-ba40-4ea8-8c7e-c36094d15bec-kube-api-access-lgd8p\" (UniqueName: \"kubernetes.io/projected/40286f5c-ba40-4ea8-8c7e-c36094d15bec-kube-api-access-lgd8p\") pod \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") "
Oct 30 19:53:56.812009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.811881    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-openshift-ca\") pod \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\" (UID: \"40286f5c-ba40-4ea8-8c7e-c36094d15bec\") "
Oct 30 19:53:56.813197 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.813153    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-openshift-ca" pod "40286f5c-ba40-4ea8-8c7e-c36094d15bec" (UID: "40286f5c-ba40-4ea8-8c7e-c36094d15bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:56.817177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.817130    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40286f5c-ba40-4ea8-8c7e-c36094d15bec-kube-api-access-lgd8p" pod "40286f5c-ba40-4ea8-8c7e-c36094d15bec" (UID: "40286f5c-ba40-4ea8-8c7e-c36094d15bec"). InnerVolumeSpecName "kube-api-access-lgd8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:56.817348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.817268    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-xks-ca" pod "40286f5c-ba40-4ea8-8c7e-c36094d15bec" (UID: "40286f5c-ba40-4ea8-8c7e-c36094d15bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:56.914724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.914677    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:56.914724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.914725    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgd8p\" (UniqueName: \"kubernetes.io/projected/40286f5c-ba40-4ea8-8c7e-c36094d15bec-kube-api-access-lgd8p\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:56.914947 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:56.914744    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40286f5c-ba40-4ea8-8c7e-c36094d15bec-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:53:57.474865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:57.474810    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f" event={"ID":"40286f5c-ba40-4ea8-8c7e-c36094d15bec","Type":"ContainerDied","Data":"f0c987eeecbb71a52b7686d4da98b590359e344a97d86ea52bc6f0e9d6e06735"}
Oct 30 19:53:57.475067 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:57.474921    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0c987eeecbb71a52b7686d4da98b590359e344a97d86ea52bc6f0e9d6e06735"
Oct 30 19:53:57.475190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:53:57.475168    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"
Oct 30 19:54:00.036238 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:00.036190    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"]
Oct 30 19:54:00.047965 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:00.047918    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc544d5-s695n"]
Oct 30 19:54:00.824744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:00.824667    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"]
Oct 30 19:54:00.834435 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:00.834271    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffc83e66-zxbwr"]
Oct 30 19:54:00.960964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:00.960913    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"]
Oct 30 19:54:00.963763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:00.961603    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2" podUID="6dc1dc0d-e491-406b-bf4c-10d0634917c5" containerName="istio-proxy" containerID="cri-o://4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4" gracePeriod=30
Oct 30 19:54:01.615326 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:01.615290    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ba08c6-7c2d-45a2-9403-12258a8f98ff" path="/var/lib/kubelet/pods/88ba08c6-7c2d-45a2-9403-12258a8f98ff/volumes"
Oct 30 19:54:01.615891 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:01.615873    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e" path="/var/lib/kubelet/pods/f53d8e9e-82a2-4154-a4fd-d0d7546bcf3e/volumes"
Oct 30 19:54:02.282720 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:54:02.279674    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:54:02.298410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.295595    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:54:02.320561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.320517    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-credential-socket\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.320737 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.320610    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-socket\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.320780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.320766    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istiod-ca-cert\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.320855 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.320829    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-credential-socket" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:02.320923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.320861    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-podinfo\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.320923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.320906    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-socket" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:02.321033 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.320934    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-certs\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.321091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321051    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-token\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-token\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.321178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321153    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istiod-ca-cert" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:02.321228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321181    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-kube-api-access-5vbst\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-kube-api-access-5vbst\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.321228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321183    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-certs" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:02.321303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321226    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-data\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.321303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-envoy\") pod \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\" (UID: \"6dc1dc0d-e491-406b-bf4c-10d0634917c5\") "
Oct 30 19:54:02.321507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321447    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-data" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:02.321955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321930    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.322059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321959    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.322059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321973    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.322059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321986    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.322059 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.321998    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.324671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.324640    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-token" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:02.324809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.324670    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-kube-api-access-5vbst" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "kube-api-access-5vbst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:02.324809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.324691    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-podinfo" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:02.327133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.327097    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-envoy" pod "6dc1dc0d-e491-406b-bf4c-10d0634917c5" (UID: "6dc1dc0d-e491-406b-bf4c-10d0634917c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:02.407796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.407707    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"]
Oct 30 19:54:02.408583 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.408555    4519 cpu_manager.go:414] "containerMap: removing container" podUID="40286f5c-ba40-4ea8-8c7e-c36094d15bec" containerName="revoke-keys"
Oct 30 19:54:02.408583 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.408582    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40286f5c-ba40-4ea8-8c7e-c36094d15bec" containerName="revoke-keys"
Oct 30 19:54:02.408789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.408650    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6dc1dc0d-e491-406b-bf4c-10d0634917c5" containerName="istio-proxy"
Oct 30 19:54:02.408789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.408657    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc1dc0d-e491-406b-bf4c-10d0634917c5" containerName="istio-proxy"
Oct 30 19:54:02.408789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.408776    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc1dc0d-e491-406b-bf4c-10d0634917c5" containerName="istio-proxy"
Oct 30 19:54:02.408789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.408790    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="40286f5c-ba40-4ea8-8c7e-c36094d15bec" containerName="revoke-keys"
Oct 30 19:54:02.413511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.413486    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.426365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.426330    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.426365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.426367    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.426596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.426381    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vbst\" (UniqueName: \"kubernetes.io/projected/6dc1dc0d-e491-406b-bf4c-10d0634917c5-kube-api-access-5vbst\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.426596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.426418    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc1dc0d-e491-406b-bf4c-10d0634917c5-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:02.435792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.435734    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-75082b11-gw-openshift-default-dockercfg-th7nq\""
Oct 30 19:54:02.443209 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.443167    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"]
Oct 30 19:54:02.527363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.527319    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-credential-socket\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.527590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.527384    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9b9\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-kube-api-access-lc9b9\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.527590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.527580    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-data\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.527726 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.527657    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187df1cd-7922-4595-b165-6489bedda57a-istio-podinfo\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.527876 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.527844    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187df1cd-7922-4595-b165-6489bedda57a-istiod-ca-cert\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.527945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.527928    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-istio-token\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.528552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.528526    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-envoy\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.528636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.528607    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-certs\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.528796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.528772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-socket\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638310    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-credential-socket\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638382    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9b9\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-kube-api-access-lc9b9\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638457    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-data\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638543    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187df1cd-7922-4595-b165-6489bedda57a-istio-podinfo\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638635    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187df1cd-7922-4595-b165-6489bedda57a-istiod-ca-cert\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638681    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-istio-token\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638711    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-envoy\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638768    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-certs\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.638902    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-socket\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.639425    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-socket\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.639724    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-credential-socket\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.640286    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-data\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.640732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-certs\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.641348    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187df1cd-7922-4595-b165-6489bedda57a-istiod-ca-cert\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.651715 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.651583    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-envoy\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.667632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.663100    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187df1cd-7922-4595-b165-6489bedda57a-istio-podinfo\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.749507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.747455    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9b9\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-kube-api-access-lc9b9\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.749507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.748172    4519 generic.go:356] "Generic (PLEG): container finished" podID="6dc1dc0d-e491-406b-bf4c-10d0634917c5" containerID="4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4" exitCode=0
Oct 30 19:54:02.749507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.748554    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2" event={"ID":"6dc1dc0d-e491-406b-bf4c-10d0634917c5","Type":"ContainerDied","Data":"4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4"}
Oct 30 19:54:02.749507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.748618    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2" event={"ID":"6dc1dc0d-e491-406b-bf4c-10d0634917c5","Type":"ContainerDied","Data":"6f590242c47f13eb41a9c556fb6ad8e76af78cb25dd4a0de9e66da655c00563b"}
Oct 30 19:54:02.749507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.748649    4519 scope.go:122] "RemoveContainer" containerID="4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4"
Oct 30 19:54:02.749507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.748872    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"
Oct 30 19:54:02.765193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.765117    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-istio-token\") pod \"e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") " pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:02.886593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.886386    4519 scope.go:122] "RemoveContainer" containerID="4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4"
Oct 30 19:54:02.887354 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:54:02.887323    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4\": container with ID starting with 4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4 not found: ID does not exist" containerID="4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4"
Oct 30 19:54:02.887606 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.887576    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4"} err="failed to get container status \"4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4\": rpc error: code = NotFound desc = could not find container \"4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4\": container with ID starting with 4af94ae354620a2ca743abc507fd469e25c40c8dab54ecaa51acf04afd35cff4 not found: ID does not exist"
Oct 30 19:54:02.899342 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.899234    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"]
Oct 30 19:54:02.951545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:02.950975    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ffc83e66-openshift-default-d65fd465f-dm6w2"]
Oct 30 19:54:03.048910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.048874    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:03.420965 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:54:03.420926    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187df1cd_7922_4595_b165_6489bedda57a.slice/crio-182b113cc7c740d8390dfc35b0ad3da41eb14b8743a9abf264ee4b8e3ceaffb1 WatchSource:0}: Error finding container 182b113cc7c740d8390dfc35b0ad3da41eb14b8743a9abf264ee4b8e3ceaffb1: Status 404 returned error can't find the container with id 182b113cc7c740d8390dfc35b0ad3da41eb14b8743a9abf264ee4b8e3ceaffb1
Oct 30 19:54:03.424129 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.424086    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:03.424251 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.424178    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:03.424251 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.424226    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:03.424638 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.424614    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"]
Oct 30 19:54:03.618698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.618656    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc1dc0d-e491-406b-bf4c-10d0634917c5" path="/var/lib/kubelet/pods/6dc1dc0d-e491-406b-bf4c-10d0634917c5/volumes"
Oct 30 19:54:03.762400 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.762305    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" event={"ID":"187df1cd-7922-4595-b165-6489bedda57a","Type":"ContainerStarted","Data":"05ad786395e636b3061fa5a313d2b437481305808b5bbbb8e17af571bbfd2b9e"}
Oct 30 19:54:03.762974 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.762953    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" event={"ID":"187df1cd-7922-4595-b165-6489bedda57a","Type":"ContainerStarted","Data":"182b113cc7c740d8390dfc35b0ad3da41eb14b8743a9abf264ee4b8e3ceaffb1"}
Oct 30 19:54:03.828553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:03.828481    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" podStartSLOduration=1.828448581 podStartE2EDuration="1.828448581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:03.821836825 +0000 UTC m=+1168.322540550" watchObservedRunningTime="2026-10-30 19:54:03.828448581 +0000 UTC m=+1168.329152306"
Oct 30 19:54:04.050239 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:04.050151    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:04.060340 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:04.060298    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:04.783498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:04.780865    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:04.795354 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:04.790156    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:54:08.495781 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.495739    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"]
Oct 30 19:54:08.500855 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.500828    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.521061 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.521028    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"]
Oct 30 19:54:08.524572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.524022    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:08.524572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.524213    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:54:08.661255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.661216    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.661537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.661507    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzzl\" (UniqueName: \"kubernetes.io/projected/8b5bf424-cd37-4f91-8deb-0e297b58e97e-kube-api-access-5vzzl\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.661618 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.661587    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.771545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.764716    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.771545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.765040    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzzl\" (UniqueName: \"kubernetes.io/projected/8b5bf424-cd37-4f91-8deb-0e297b58e97e-kube-api-access-5vzzl\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.771545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.765123    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.771545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.766107    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.771545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.770799    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.787397 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.787366    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzzl\" (UniqueName: \"kubernetes.io/projected/8b5bf424-cd37-4f91-8deb-0e297b58e97e-kube-api-access-5vzzl\") pod \"maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:08.820333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:08.820296    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:09.041184 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.041092    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"]
Oct 30 19:54:09.053650 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.053578    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a0c2c0-kqw9p"]
Oct 30 19:54:09.077210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.077161    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"]
Oct 30 19:54:09.083242 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.083208    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.114220 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.113138    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"]
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179057    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179213    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179328    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9045b774-bcc5-4441-bf11-8c3dba36f363-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179482    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5wgf\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-kube-api-access-h5wgf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179534    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179583    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179615    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179653    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.186121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.179698    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.244416 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.244375    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"]
Oct 30 19:54:09.281324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281287    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9045b774-bcc5-4441-bf11-8c3dba36f363-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.281555 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281484    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wgf\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-kube-api-access-h5wgf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.281613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281563    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.281664 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.281664 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281650    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.281771 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281679    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.281771 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281728    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.281874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.281856    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.282037 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.282019    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.282110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.282095    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.282165 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.282091    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.282262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.282241    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9045b774-bcc5-4441-bf11-8c3dba36f363-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.282394 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.282365    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.282573 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.282397    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.284331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.284304    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.284665 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.284641    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.301460 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.301381    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.301620 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.301494    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wgf\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-kube-api-access-h5wgf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c2xqx\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.419036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.418997    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:09.615841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.615758    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a15706-2d2b-402f-bbe7-860bf13f8264" path="/var/lib/kubelet/pods/27a15706-2d2b-402f-bbe7-860bf13f8264/volumes"
Oct 30 19:54:09.760317 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:54:09.760285    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9045b774_bcc5_4441_bf11_8c3dba36f363.slice/crio-b30ab067d086f068b2f9b72c537f9fdbf10b78ae071907c4553102f490dbfae2 WatchSource:0}: Error finding container b30ab067d086f068b2f9b72c537f9fdbf10b78ae071907c4553102f490dbfae2: Status 404 returned error can't find the container with id b30ab067d086f068b2f9b72c537f9fdbf10b78ae071907c4553102f490dbfae2
Oct 30 19:54:09.762373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.762346    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"]
Oct 30 19:54:09.763599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.763565    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:09.763691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.763641    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:09.763691 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.763670    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:09.878178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.878074    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5" event={"ID":"8b5bf424-cd37-4f91-8deb-0e297b58e97e","Type":"ContainerStarted","Data":"0498430f27f0ce8727e1735032b2eae2143c1d721aeafd155f7baf6068736d2a"}
Oct 30 19:54:09.878178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.878151    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5" event={"ID":"8b5bf424-cd37-4f91-8deb-0e297b58e97e","Type":"ContainerStarted","Data":"05d2b99cf7c0ed14266547c04974420457f041007089ae975df6bbacf6b5220e"}
Oct 30 19:54:09.880829 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.880788    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" event={"ID":"9045b774-bcc5-4441-bf11-8c3dba36f363","Type":"ContainerStarted","Data":"0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144"}
Oct 30 19:54:09.881006 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.880865    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" event={"ID":"9045b774-bcc5-4441-bf11-8c3dba36f363","Type":"ContainerStarted","Data":"b30ab067d086f068b2f9b72c537f9fdbf10b78ae071907c4553102f490dbfae2"}
Oct 30 19:54:09.921308 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.921253    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5" podStartSLOduration=1.921238907 podStartE2EDuration="1.921238907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:09.920146215 +0000 UTC m=+1174.420849933" watchObservedRunningTime="2026-10-30 19:54:09.921238907 +0000 UTC m=+1174.421942631"
Oct 30 19:54:09.973905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:09.973843    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podStartSLOduration=0.973826606 podStartE2EDuration="973.826606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:09.973333047 +0000 UTC m=+1174.474036768" watchObservedRunningTime="2026-10-30 19:54:09.973826606 +0000 UTC m=+1174.474530330"
Oct 30 19:54:10.035432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:10.035379    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"]
Oct 30 19:54:10.049589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:10.049545    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d42b0ec6-kjd2l"]
Oct 30 19:54:10.420189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:10.420143    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:10.422275 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:10.422243    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:10.422350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:10.422302    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:10.931605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:10.931562    4519 generic.go:356] "Generic (PLEG): container finished" podID="8b5bf424-cd37-4f91-8deb-0e297b58e97e" containerID="0498430f27f0ce8727e1735032b2eae2143c1d721aeafd155f7baf6068736d2a" exitCode=0
Oct 30 19:54:10.933944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:10.933909    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5" event={"ID":"8b5bf424-cd37-4f91-8deb-0e297b58e97e","Type":"ContainerDied","Data":"0498430f27f0ce8727e1735032b2eae2143c1d721aeafd155f7baf6068736d2a"}
Oct 30 19:54:11.174607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:11.174527    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"]
Oct 30 19:54:11.258664 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:11.242092    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d61bf1d-tvbh9"]
Oct 30 19:54:11.421016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:11.420973    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:11.421220 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:11.421048    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:11.841500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:11.840248    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eabf850-bde4-4d95-8740-2936222afb33" path="/var/lib/kubelet/pods/8eabf850-bde4-4d95-8740-2936222afb33/volumes"
Oct 30 19:54:11.841500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:11.840948    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd6f340-f243-4dc1-b425-afa89d505e69" path="/var/lib/kubelet/pods/abd6f340-f243-4dc1-b425-afa89d505e69/volumes"
Oct 30 19:54:12.036243 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.036201    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"]
Oct 30 19:54:12.076650 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.046459    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cece07-2scnn"]
Oct 30 19:54:12.190121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.190089    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"]
Oct 30 19:54:12.196898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.196865    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b58e9f2-l886f"]
Oct 30 19:54:12.232788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.232749    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:12.338030 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.337986    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-openshift-ca\") pod \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") "
Oct 30 19:54:12.338198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.338129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5bf424-cd37-4f91-8deb-0e297b58e97e-kube-api-access-5vzzl\" (UniqueName: \"kubernetes.io/projected/8b5bf424-cd37-4f91-8deb-0e297b58e97e-kube-api-access-5vzzl\") pod \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") "
Oct 30 19:54:12.338198 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.338190    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-xks-ca\") pod \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\" (UID: \"8b5bf424-cd37-4f91-8deb-0e297b58e97e\") "
Oct 30 19:54:12.342980 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.342935    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-openshift-ca" pod "8b5bf424-cd37-4f91-8deb-0e297b58e97e" (UID: "8b5bf424-cd37-4f91-8deb-0e297b58e97e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:12.396181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.396138    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5bf424-cd37-4f91-8deb-0e297b58e97e-kube-api-access-5vzzl" pod "8b5bf424-cd37-4f91-8deb-0e297b58e97e" (UID: "8b5bf424-cd37-4f91-8deb-0e297b58e97e"). InnerVolumeSpecName "kube-api-access-5vzzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:12.396348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.396233    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-xks-ca" pod "8b5bf424-cd37-4f91-8deb-0e297b58e97e" (UID: "8b5bf424-cd37-4f91-8deb-0e297b58e97e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:12.420751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.420609    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:12.420751 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.420679    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:12.440842 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.440755    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:12.440842 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.440786    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vzzl\" (UniqueName: \"kubernetes.io/projected/8b5bf424-cd37-4f91-8deb-0e297b58e97e-kube-api-access-5vzzl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:12.440842 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:12.440801    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bf424-cd37-4f91-8deb-0e297b58e97e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:13.004200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:13.004154    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5" event={"ID":"8b5bf424-cd37-4f91-8deb-0e297b58e97e","Type":"ContainerDied","Data":"05d2b99cf7c0ed14266547c04974420457f041007089ae975df6bbacf6b5220e"}
Oct 30 19:54:13.004522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:13.004500    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d2b99cf7c0ed14266547c04974420457f041007089ae975df6bbacf6b5220e"
Oct 30 19:54:13.004680 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:13.004191    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"
Oct 30 19:54:13.419719 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:13.419631    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:13.420153 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:13.419818    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:13.616240 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:13.616196    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40286f5c-ba40-4ea8-8c7e-c36094d15bec" path="/var/lib/kubelet/pods/40286f5c-ba40-4ea8-8c7e-c36094d15bec/volumes"
Oct 30 19:54:13.616998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:13.616971    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e6279a-f362-4067-bcdb-59b369f04c71" path="/var/lib/kubelet/pods/77e6279a-f362-4067-bcdb-59b369f04c71/volumes"
Oct 30 19:54:14.108384 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:14.108302    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"]
Oct 30 19:54:14.157910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:14.157872    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ade06a-mnfmm"]
Oct 30 19:54:14.419706 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:14.419667    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:14.419894 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:14.419726    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:15.035436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:15.035398    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"]
Oct 30 19:54:15.052492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:15.046949    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da3fe3-q8k74"]
Oct 30 19:54:15.420061 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:15.420026    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:15.420502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:15.420113    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:15.615363 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:15.615328    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799bdb9d-dd16-4e35-b92f-fe2699054515" path="/var/lib/kubelet/pods/799bdb9d-dd16-4e35-b92f-fe2699054515/volumes"
Oct 30 19:54:15.615851 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:15.615835    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f171125-a5d1-4a3a-b6d5-8b56531ef647" path="/var/lib/kubelet/pods/9f171125-a5d1-4a3a-b6d5-8b56531ef647/volumes"
Oct 30 19:54:16.035756 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:16.035716    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"]
Oct 30 19:54:16.063944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:16.063899    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ce72fc-k8g6q"]
Oct 30 19:54:16.420367 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:16.420323    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:16.420976 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:16.420387    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:17.420066 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:17.420022    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:54:17.420297 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:17.420110    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:54:17.614234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:17.614186    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95c3e3a-6249-4907-9252-e04dc15ce93a" path="/var/lib/kubelet/pods/e95c3e3a-6249-4907-9252-e04dc15ce93a/volumes"
Oct 30 19:54:19.448491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:19.437110    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:19.448491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:19.437195    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:19.694697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:19.694659    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:19.696411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:19.696374    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:19.698046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:19.698018    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 19:54:22.224013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.223969    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"]
Oct 30 19:54:22.234025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.233982    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc2ad28e-76fh5"]
Oct 30 19:54:22.800931 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.800891    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"]
Oct 30 19:54:22.801621 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.801604    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8b5bf424-cd37-4f91-8deb-0e297b58e97e" containerName="revoke-keys"
Oct 30 19:54:22.801669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.801623    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5bf424-cd37-4f91-8deb-0e297b58e97e" containerName="revoke-keys"
Oct 30 19:54:22.801741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.801732    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5bf424-cd37-4f91-8deb-0e297b58e97e" containerName="revoke-keys"
Oct 30 19:54:22.806971 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.806945    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:22.826527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.826490    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"]
Oct 30 19:54:22.839853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.839810    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:54:22.840111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.839935    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:22.938804 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.938762    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:22.939035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.939006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzh96\" (UniqueName: \"kubernetes.io/projected/7688d89d-62cf-4c58-8543-451cc668293b-kube-api-access-jzh96\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:22.939177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:22.939161    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.081567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.075707    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzh96\" (UniqueName: \"kubernetes.io/projected/7688d89d-62cf-4c58-8543-451cc668293b-kube-api-access-jzh96\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.081567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.075873    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.081567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.076036    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.081567 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.078096    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.174487 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.170429    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.179741 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.179699    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzh96\" (UniqueName: \"kubernetes.io/projected/7688d89d-62cf-4c58-8543-451cc668293b-kube-api-access-jzh96\") pod \"maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.424752 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.424710    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:23.614680 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.614644    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5bf424-cd37-4f91-8deb-0e297b58e97e" path="/var/lib/kubelet/pods/8b5bf424-cd37-4f91-8deb-0e297b58e97e/volumes"
Oct 30 19:54:23.746659 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:54:23.746616    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7688d89d_62cf_4c58_8543_451cc668293b.slice/crio-4c8df74b141704e041a01078738ee5d253507820410ef951c84179141e9a7e1f WatchSource:0}: Error finding container 4c8df74b141704e041a01078738ee5d253507820410ef951c84179141e9a7e1f: Status 404 returned error can't find the container with id 4c8df74b141704e041a01078738ee5d253507820410ef951c84179141e9a7e1f
Oct 30 19:54:23.751092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:23.751059    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"]
Oct 30 19:54:24.045552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.045451    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"]
Oct 30 19:54:24.057056 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.057023    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"]
Oct 30 19:54:24.066607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.066537    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a52caf0-f7vmg"]
Oct 30 19:54:24.076768 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.076718    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f068df-2k2nk"]
Oct 30 19:54:24.087068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.087021    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"]
Oct 30 19:54:24.098920 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.097654    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.152748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.149559    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"]
Oct 30 19:54:24.222062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.222016    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.222244 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.222122    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.222244 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.222223    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.222365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.222282    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfjsw\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-kube-api-access-cfjsw\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.222435 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.222412    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668957ef-930c-4e75-a6f2-3ee872c21db4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.222587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.222484    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.222649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.222582    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.224950 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.224909    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.225118 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.224971    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.283072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.283016    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd" event={"ID":"7688d89d-62cf-4c58-8543-451cc668293b","Type":"ContainerStarted","Data":"40e2a0d058a5f55aa903a5b280dbd2b32339804cadbed13bf9143ecdb8a98769"}
Oct 30 19:54:24.283546 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.283107    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd" event={"ID":"7688d89d-62cf-4c58-8543-451cc668293b","Type":"ContainerStarted","Data":"4c8df74b141704e041a01078738ee5d253507820410ef951c84179141e9a7e1f"}
Oct 30 19:54:24.328910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.328812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.329213 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329189    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.329417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329400    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.329610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329593    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfjsw\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-kube-api-access-cfjsw\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.329755 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329738    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668957ef-930c-4e75-a6f2-3ee872c21db4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.329840 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329791    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.329840 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329795    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.329947 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329916    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.330002 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.329976    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.330052 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.330017    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.330213 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.330189    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.330435 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.330417    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.330596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.330456    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668957ef-930c-4e75-a6f2-3ee872c21db4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.330702 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.330681    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.333260 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.333208    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.335449 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.335396    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.337307 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.337259    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd" podStartSLOduration=2.337243042 podStartE2EDuration="2.337243042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:24.328006386 +0000 UTC m=+1188.828710110" watchObservedRunningTime="2026-10-30 19:54:24.337243042 +0000 UTC m=+1188.837946765"
Oct 30 19:54:24.354086 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.354046    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.354405 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.354373    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfjsw\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-kube-api-access-cfjsw\") pod \"maas-default-gateway-openshift-default-75bf867b6c-kzgzl\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:24.497958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:24.497924    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:25.198578 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.198459    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:25.198783 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.198700    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:25.198856 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.198791    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:25.231702 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.231513    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"]
Oct 30 19:54:25.319521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.319461    4519 generic.go:356] "Generic (PLEG): container finished" podID="7688d89d-62cf-4c58-8543-451cc668293b" containerID="40e2a0d058a5f55aa903a5b280dbd2b32339804cadbed13bf9143ecdb8a98769" exitCode=0
Oct 30 19:54:25.320001 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.319971    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd" event={"ID":"7688d89d-62cf-4c58-8543-451cc668293b","Type":"ContainerDied","Data":"40e2a0d058a5f55aa903a5b280dbd2b32339804cadbed13bf9143ecdb8a98769"}
Oct 30 19:54:25.333495 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.324032    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" event={"ID":"668957ef-930c-4e75-a6f2-3ee872c21db4","Type":"ContainerStarted","Data":"c3d4d2f2157fb76ee88505a4ca1fd3d90f914044ff2af241fcc801f3bc659ecb"}
Oct 30 19:54:25.820489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.819593    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09003d52-027e-4859-8e6e-52b9e0c15daa" path="/var/lib/kubelet/pods/09003d52-027e-4859-8e6e-52b9e0c15daa/volumes"
Oct 30 19:54:25.820489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:25.820368    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc7717d-3b98-4b2b-bf93-2d6194a73452" path="/var/lib/kubelet/pods/3bc7717d-3b98-4b2b-bf93-2d6194a73452/volumes"
Oct 30 19:54:26.054625 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.054588    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"]
Oct 30 19:54:26.082558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.064282    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"]
Oct 30 19:54:26.112536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.109081    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fb68e6-f27kj"]
Oct 30 19:54:26.119541 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.118399    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2a96a4-b9nhr"]
Oct 30 19:54:26.345823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.345722    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" event={"ID":"668957ef-930c-4e75-a6f2-3ee872c21db4","Type":"ContainerStarted","Data":"1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd"}
Oct 30 19:54:26.392182 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.392114    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podStartSLOduration=2.392092542 podStartE2EDuration="2.392092542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:26.390888513 +0000 UTC m=+1190.891592253" watchObservedRunningTime="2026-10-30 19:54:26.392092542 +0000 UTC m=+1190.892796268"
Oct 30 19:54:26.499333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.499276    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:26.502967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.502049    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:26.502967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.502331    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:26.617028 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.616951    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:26.731937 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.729852    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-xks-ca\") pod \"7688d89d-62cf-4c58-8543-451cc668293b\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") "
Oct 30 19:54:26.731937 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.730103    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7688d89d-62cf-4c58-8543-451cc668293b-kube-api-access-jzh96\" (UniqueName: \"kubernetes.io/projected/7688d89d-62cf-4c58-8543-451cc668293b-kube-api-access-jzh96\") pod \"7688d89d-62cf-4c58-8543-451cc668293b\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") "
Oct 30 19:54:26.731937 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.730211    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-openshift-ca\") pod \"7688d89d-62cf-4c58-8543-451cc668293b\" (UID: \"7688d89d-62cf-4c58-8543-451cc668293b\") "
Oct 30 19:54:26.733596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.733561    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-openshift-ca" pod "7688d89d-62cf-4c58-8543-451cc668293b" (UID: "7688d89d-62cf-4c58-8543-451cc668293b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:26.734372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.734341    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7688d89d-62cf-4c58-8543-451cc668293b-kube-api-access-jzh96" pod "7688d89d-62cf-4c58-8543-451cc668293b" (UID: "7688d89d-62cf-4c58-8543-451cc668293b"). InnerVolumeSpecName "kube-api-access-jzh96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:26.742410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.742367    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-xks-ca" pod "7688d89d-62cf-4c58-8543-451cc668293b" (UID: "7688d89d-62cf-4c58-8543-451cc668293b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:26.833241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.833198    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:26.833241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.833234    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7688d89d-62cf-4c58-8543-451cc668293b-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:26.833241 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:26.833244    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzh96\" (UniqueName: \"kubernetes.io/projected/7688d89d-62cf-4c58-8543-451cc668293b-kube-api-access-jzh96\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:27.498192 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:27.496493    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"
Oct 30 19:54:27.520501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:27.520045    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd" event={"ID":"7688d89d-62cf-4c58-8543-451cc668293b","Type":"ContainerDied","Data":"4c8df74b141704e041a01078738ee5d253507820410ef951c84179141e9a7e1f"}
Oct 30 19:54:27.520501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:27.520122    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c8df74b141704e041a01078738ee5d253507820410ef951c84179141e9a7e1f"
Oct 30 19:54:27.526703 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:27.526650    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:27.527020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:27.526971    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:27.673593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:27.672952    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e2ac64-9211-48c5-9b37-e748f83b9f45" path="/var/lib/kubelet/pods/95e2ac64-9211-48c5-9b37-e748f83b9f45/volumes"
Oct 30 19:54:27.719491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:27.714910    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f378ee03-dabe-4875-aaa1-104d8af75689" path="/var/lib/kubelet/pods/f378ee03-dabe-4875-aaa1-104d8af75689/volumes"
Oct 30 19:54:28.503456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:28.503415    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:28.504008 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:28.503503    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:29.038972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:29.038927    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"]
Oct 30 19:54:29.062493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:29.060813    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f55efe-bxktb"]
Oct 30 19:54:29.499653 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:29.499603    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:29.499859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:29.499674    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:29.615796 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:29.615758    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ce29d4-49e9-438d-beca-1364d75a7bd9" path="/var/lib/kubelet/pods/10ce29d4-49e9-438d-beca-1364d75a7bd9/volumes"
Oct 30 19:54:30.500528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:30.500485    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:30.500799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:30.500771    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:31.506020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:31.505976    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:31.514632 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:31.514574    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:32.223192 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:54:32.223150    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:54:32.499502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:32.499364    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:32.499502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:32.499447    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:33.499682 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:33.499610    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:33.500228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:33.499703    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:34.070236 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:34.070192    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"]
Oct 30 19:54:34.082421 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:34.082286    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cfbff9-qggvg"]
Oct 30 19:54:34.499368 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:34.499324    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-kzgzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:54:34.499613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:34.499384    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:54:35.618313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:35.618264    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78851ec-275d-4750-8e84-4a3f0e16bc85" path="/var/lib/kubelet/pods/e78851ec-275d-4750-8e84-4a3f0e16bc85/volumes"
Oct 30 19:54:36.117667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:36.117632    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:36.118824 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:36.118798    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:36.119676 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:36.119647    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 19:54:37.077942 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.077897    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"]
Oct 30 19:54:37.078835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.078810    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7688d89d-62cf-4c58-8543-451cc668293b" containerName="revoke-keys"
Oct 30 19:54:37.078953 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.078835    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7688d89d-62cf-4c58-8543-451cc668293b" containerName="revoke-keys"
Oct 30 19:54:37.079049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.079035    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7688d89d-62cf-4c58-8543-451cc668293b" containerName="revoke-keys"
Oct 30 19:54:37.084507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.084442    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.107528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.106743    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"]
Oct 30 19:54:37.108334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.108308    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:37.108579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.108356    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:54:37.167016 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.166978    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.167247 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.167146    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8cf\" (UniqueName: \"kubernetes.io/projected/dbd820d2-d4d4-4002-8843-1136087a44ab-kube-api-access-vh8cf\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.167326 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.167288    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.269326 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.269282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.269596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.269423    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.269678 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.269595    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8cf\" (UniqueName: \"kubernetes.io/projected/dbd820d2-d4d4-4002-8843-1136087a44ab-kube-api-access-vh8cf\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.270116 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.270089    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.272168 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.272143    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.289930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.289896    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8cf\" (UniqueName: \"kubernetes.io/projected/dbd820d2-d4d4-4002-8843-1136087a44ab-kube-api-access-vh8cf\") pod \"maas-api-revoke-sub-8308e4b9-d5862\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.402951 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.402852    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:37.718359 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:54:37.718314    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd820d2_d4d4_4002_8843_1136087a44ab.slice/crio-d6127100050cfefbda90840d481dae3804e8cdbea3eb490c097cd89273650cc2 WatchSource:0}: Error finding container d6127100050cfefbda90840d481dae3804e8cdbea3eb490c097cd89273650cc2: Status 404 returned error can't find the container with id d6127100050cfefbda90840d481dae3804e8cdbea3eb490c097cd89273650cc2
Oct 30 19:54:37.721898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:37.721869    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"]
Oct 30 19:54:38.787485 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:38.787417    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862" event={"ID":"dbd820d2-d4d4-4002-8843-1136087a44ab","Type":"ContainerStarted","Data":"b5f56586f066cc19fe54b617c8d2f740dff56da56495597ed1d67621dd61834d"}
Oct 30 19:54:38.788125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:38.788096    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862" event={"ID":"dbd820d2-d4d4-4002-8843-1136087a44ab","Type":"ContainerStarted","Data":"d6127100050cfefbda90840d481dae3804e8cdbea3eb490c097cd89273650cc2"}
Oct 30 19:54:38.920488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:38.877532    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862" podStartSLOduration=1.8775051029999998 podStartE2EDuration="1.877505103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:38.841857894 +0000 UTC m=+1203.342561619" watchObservedRunningTime="2026-10-30 19:54:38.877505103 +0000 UTC m=+1203.378208828"
Oct 30 19:54:38.925739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:38.925680    4519 scope.go:122] "RemoveContainer" containerID="ccebcf7a0f27875310bb0c88ed8c63c18b6b92037c9db38effea2a09633b2868"
Oct 30 19:54:38.963185 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:38.960540    4519 scope.go:122] "RemoveContainer" containerID="a3a10093ac4c8fc5f7d0ea12dc5662496d6cd8dca396e79f8ba72a038283eaf0"
Oct 30 19:54:38.997338 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:38.997293    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"]
Oct 30 19:54:39.006628 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:39.006587    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9c6d81d-n2vfd"]
Oct 30 19:54:39.019609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:39.019572    4519 scope.go:122] "RemoveContainer" containerID="e8c387b5628fd834c5973c489b297cdea99d726505c947d1bec322432607accb"
Oct 30 19:54:39.623534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:39.623283    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7688d89d-62cf-4c58-8543-451cc668293b" path="/var/lib/kubelet/pods/7688d89d-62cf-4c58-8543-451cc668293b/volumes"
Oct 30 19:54:39.800274 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:39.800233    4519 generic.go:356] "Generic (PLEG): container finished" podID="dbd820d2-d4d4-4002-8843-1136087a44ab" containerID="b5f56586f066cc19fe54b617c8d2f740dff56da56495597ed1d67621dd61834d" exitCode=0
Oct 30 19:54:39.800750 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:39.800317    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862" event={"ID":"dbd820d2-d4d4-4002-8843-1136087a44ab","Type":"ContainerDied","Data":"b5f56586f066cc19fe54b617c8d2f740dff56da56495597ed1d67621dd61834d"}
Oct 30 19:54:41.022503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.015350    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:41.055911 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.045518    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"]
Oct 30 19:54:41.066558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.063448    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2c89ee-9gvnb"]
Oct 30 19:54:41.093735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.089401    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd820d2-d4d4-4002-8843-1136087a44ab-kube-api-access-vh8cf\" (UniqueName: \"kubernetes.io/projected/dbd820d2-d4d4-4002-8843-1136087a44ab-kube-api-access-vh8cf\") pod \"dbd820d2-d4d4-4002-8843-1136087a44ab\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") "
Oct 30 19:54:41.093735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.090258    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-openshift-ca\") pod \"dbd820d2-d4d4-4002-8843-1136087a44ab\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") "
Oct 30 19:54:41.093735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.090776    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-openshift-ca" pod "dbd820d2-d4d4-4002-8843-1136087a44ab" (UID: "dbd820d2-d4d4-4002-8843-1136087a44ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:41.093735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.090963    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-xks-ca\") pod \"dbd820d2-d4d4-4002-8843-1136087a44ab\" (UID: \"dbd820d2-d4d4-4002-8843-1136087a44ab\") "
Oct 30 19:54:41.093735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.092173    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:41.114552 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.099664    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-xks-ca" pod "dbd820d2-d4d4-4002-8843-1136087a44ab" (UID: "dbd820d2-d4d4-4002-8843-1136087a44ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:41.119312 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.119268    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd820d2-d4d4-4002-8843-1136087a44ab-kube-api-access-vh8cf" pod "dbd820d2-d4d4-4002-8843-1136087a44ab" (UID: "dbd820d2-d4d4-4002-8843-1136087a44ab"). InnerVolumeSpecName "kube-api-access-vh8cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:41.199544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.195061    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8cf\" (UniqueName: \"kubernetes.io/projected/dbd820d2-d4d4-4002-8843-1136087a44ab-kube-api-access-vh8cf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:41.199544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.195120    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd820d2-d4d4-4002-8843-1136087a44ab-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:41.614784 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.614750    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acce9ca9-bca5-4932-9ceb-35cc4bd63055" path="/var/lib/kubelet/pods/acce9ca9-bca5-4932-9ceb-35cc4bd63055/volumes"
Oct 30 19:54:41.822559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.822531    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"
Oct 30 19:54:41.822742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.822534    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862" event={"ID":"dbd820d2-d4d4-4002-8843-1136087a44ab","Type":"ContainerDied","Data":"d6127100050cfefbda90840d481dae3804e8cdbea3eb490c097cd89273650cc2"}
Oct 30 19:54:41.822742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:41.822657    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6127100050cfefbda90840d481dae3804e8cdbea3eb490c097cd89273650cc2"
Oct 30 19:54:43.039147 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:43.039100    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"]
Oct 30 19:54:43.053733 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:43.053689    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f800c7-5f6qt"]
Oct 30 19:54:43.614989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:43.614954    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c" path="/var/lib/kubelet/pods/d6a5a9c1-0c34-444d-97a3-d1f6bb2c846c/volumes"
Oct 30 19:54:46.396139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.394301    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"]
Oct 30 19:54:46.396139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.395423    4519 cpu_manager.go:414] "containerMap: removing container" podUID="dbd820d2-d4d4-4002-8843-1136087a44ab" containerName="revoke-keys"
Oct 30 19:54:46.396139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.395450    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd820d2-d4d4-4002-8843-1136087a44ab" containerName="revoke-keys"
Oct 30 19:54:46.396139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.395675    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd820d2-d4d4-4002-8843-1136087a44ab" containerName="revoke-keys"
Oct 30 19:54:46.406344 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.405866    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.412679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.412642    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"]
Oct 30 19:54:46.434712 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.434680    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:54:46.435017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.434959    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:46.501930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.501886    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.502177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.502046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.502262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.502176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q77z\" (UniqueName: \"kubernetes.io/projected/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-kube-api-access-5q77z\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.604146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.604098    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.604362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.604207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.604362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.604307    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q77z\" (UniqueName: \"kubernetes.io/projected/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-kube-api-access-5q77z\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.605150 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.605122    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.608703 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.608670    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.634295 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.634244    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q77z\" (UniqueName: \"kubernetes.io/projected/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-kube-api-access-5q77z\") pod \"maas-api-revoke-keys-e2e-migrate-75082b11-26pvc\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:46.739046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:46.739008    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:47.213388 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:47.213339    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"]
Oct 30 19:54:47.915009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:47.914957    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc" event={"ID":"050d1f27-0d29-4591-9ea0-aa9b6ae38f65","Type":"ContainerStarted","Data":"d46e7761f48d80e0ed336a87c1203b2c4b7140c4bc23d35a36b3e817108eb8ac"}
Oct 30 19:54:47.915587 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:47.915080    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc" event={"ID":"050d1f27-0d29-4591-9ea0-aa9b6ae38f65","Type":"ContainerStarted","Data":"ea7ade87f865bfaca6680eed6016ed9af15d85a8a2ba4c9b440a76f9e4e76f70"}
Oct 30 19:54:47.995730 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:47.991690    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc" podStartSLOduration=1.991665713 podStartE2EDuration="1.991665713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:47.95502393 +0000 UTC m=+1212.455727655" watchObservedRunningTime="2026-10-30 19:54:47.991665713 +0000 UTC m=+1212.492369438"
Oct 30 19:54:48.946436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:48.946396    4519 generic.go:356] "Generic (PLEG): container finished" podID="050d1f27-0d29-4591-9ea0-aa9b6ae38f65" containerID="d46e7761f48d80e0ed336a87c1203b2c4b7140c4bc23d35a36b3e817108eb8ac" exitCode=0
Oct 30 19:54:48.947175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:48.947142    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc" event={"ID":"050d1f27-0d29-4591-9ea0-aa9b6ae38f65","Type":"ContainerDied","Data":"d46e7761f48d80e0ed336a87c1203b2c4b7140c4bc23d35a36b3e817108eb8ac"}
Oct 30 19:54:50.970532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:50.923329    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.033217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.021452    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c2xqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:51.033217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.021538    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.092498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.087146    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.116111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.112494    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.116111 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.114567    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:54:51.130147 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.127133    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.148053 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.147011    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.148053 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.147251    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:54:51.148053 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.147424    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:54:51.206011 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.205970    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:51.345433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.345327    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-xks-ca\") pod \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") "
Oct 30 19:54:51.345707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.345682    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-kube-api-access-5q77z\" (UniqueName: \"kubernetes.io/projected/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-kube-api-access-5q77z\") pod \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") "
Oct 30 19:54:51.346049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.346021    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-openshift-ca\") pod \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\" (UID: \"050d1f27-0d29-4591-9ea0-aa9b6ae38f65\") "
Oct 30 19:54:51.355657 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.355598    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-openshift-ca" pod "050d1f27-0d29-4591-9ea0-aa9b6ae38f65" (UID: "050d1f27-0d29-4591-9ea0-aa9b6ae38f65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:51.387789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.372824    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-xks-ca" pod "050d1f27-0d29-4591-9ea0-aa9b6ae38f65" (UID: "050d1f27-0d29-4591-9ea0-aa9b6ae38f65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:51.387789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.386367    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-kube-api-access-5q77z" pod "050d1f27-0d29-4591-9ea0-aa9b6ae38f65" (UID: "050d1f27-0d29-4591-9ea0-aa9b6ae38f65"). InnerVolumeSpecName "kube-api-access-5q77z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:51.460954 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.460915    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:51.461193 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.461172    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q77z\" (UniqueName: \"kubernetes.io/projected/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-kube-api-access-5q77z\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:51.463448 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:51.463394    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050d1f27-0d29-4591-9ea0-aa9b6ae38f65-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:54:52.342295 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:52.342139    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc" event={"ID":"050d1f27-0d29-4591-9ea0-aa9b6ae38f65","Type":"ContainerDied","Data":"ea7ade87f865bfaca6680eed6016ed9af15d85a8a2ba4c9b440a76f9e4e76f70"}
Oct 30 19:54:52.342999 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:52.342973    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea7ade87f865bfaca6680eed6016ed9af15d85a8a2ba4c9b440a76f9e4e76f70"
Oct 30 19:54:52.343226 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:52.343208    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"
Oct 30 19:54:59.521633 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.521595    4519 generic.go:356] "Generic (PLEG): container finished" podID="187df1cd-7922-4595-b165-6489bedda57a" containerID="05ad786395e636b3061fa5a313d2b437481305808b5bbbb8e17af571bbfd2b9e" exitCode=0
Oct 30 19:54:59.522096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.521672    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" event={"ID":"187df1cd-7922-4595-b165-6489bedda57a","Type":"ContainerDied","Data":"05ad786395e636b3061fa5a313d2b437481305808b5bbbb8e17af571bbfd2b9e"}
Oct 30 19:54:59.522180 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.522165    4519 scope.go:122] "RemoveContainer" containerID="05ad786395e636b3061fa5a313d2b437481305808b5bbbb8e17af571bbfd2b9e"
Oct 30 19:54:59.523816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.523783    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:59.523941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.523873    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:59.523941 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.523926    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:59.996005 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.995964    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"]
Oct 30 19:54:59.996700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.996684    4519 cpu_manager.go:414] "containerMap: removing container" podUID="050d1f27-0d29-4591-9ea0-aa9b6ae38f65" containerName="revoke-keys"
Oct 30 19:54:59.996767 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.996701    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050d1f27-0d29-4591-9ea0-aa9b6ae38f65" containerName="revoke-keys"
Oct 30 19:54:59.996810 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:54:59.996803    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="050d1f27-0d29-4591-9ea0-aa9b6ae38f65" containerName="revoke-keys"
Oct 30 19:55:00.000821 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.000791    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.022909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.022864    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"]
Oct 30 19:55:00.074554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.074521    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:00.074777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.074635    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:55:00.126876 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.126839    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.127082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.126981    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.127082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.127069    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qbk\" (UniqueName: \"kubernetes.io/projected/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-kube-api-access-w9qbk\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.228777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.228735    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.228978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.228844    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.228978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.228905    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qbk\" (UniqueName: \"kubernetes.io/projected/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-kube-api-access-w9qbk\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.229603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.229576    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.231454 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.231431    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.250106 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.250022    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qbk\" (UniqueName: \"kubernetes.io/projected/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-kube-api-access-w9qbk\") pod \"maas-api-revoke-sub-ce8de508-6xksx\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.317042 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.317003    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:00.635549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.634004    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" event={"ID":"187df1cd-7922-4595-b165-6489bedda57a","Type":"ContainerStarted","Data":"14d52b7d681427d8cfbee9207df09388eceaaba88b5212ffec8dcd02b78ea0b4"}
Oct 30 19:55:00.791487 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:55:00.771147    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24c6ec9_82a9_4e4c_bec7_630f35e0cb7b.slice/crio-1832b04d942684494b9b125dc218a958a3df9d3a2282e848bc85100e5cd09101 WatchSource:0}: Error finding container 1832b04d942684494b9b125dc218a958a3df9d3a2282e848bc85100e5cd09101: Status 404 returned error can't find the container with id 1832b04d942684494b9b125dc218a958a3df9d3a2282e848bc85100e5cd09101
Oct 30 19:55:00.811491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:00.803152    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"]
Oct 30 19:55:01.051246 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:01.051211    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:55:01.051699 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:01.051669    4519 patch_prober.go:29] interesting pod/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:55:01.051919 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:01.051892    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 19:55:01.419674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:01.419612    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"]
Oct 30 19:55:01.439893 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:01.439847    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-75082b11-26pvc"]
Oct 30 19:55:01.534844 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:01.534591    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"]
Oct 30 19:55:01.992508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:01.982186    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050d1f27-0d29-4591-9ea0-aa9b6ae38f65" path="/var/lib/kubelet/pods/050d1f27-0d29-4591-9ea0-aa9b6ae38f65/volumes"
Oct 30 19:55:02.037961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:02.037914    4519 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" secret="" err="secret \"e2e-migrate-75082b11-gw-openshift-default-dockercfg-th7nq\" not found"
Oct 30 19:55:02.039187 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:02.039150    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx" event={"ID":"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b","Type":"ContainerStarted","Data":"b409047454d51999e63bf505a6e27edd906a457c829771ab9419a8dd4198e93d"}
Oct 30 19:55:02.039345 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:02.039197    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx" event={"ID":"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b","Type":"ContainerStarted","Data":"1832b04d942684494b9b125dc218a958a3df9d3a2282e848bc85100e5cd09101"}
Oct 30 19:55:02.088554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:02.088510    4519 patch_prober.go:29] interesting pod/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:55:02.088775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:02.088580    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 19:55:02.126168 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:02.125985    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx" podStartSLOduration=3.125970175 podStartE2EDuration="3.125970175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:02.115817653 +0000 UTC m=+1226.616521378" watchObservedRunningTime="2026-10-30 19:55:02.125970175 +0000 UTC m=+1226.626673899"
Oct 30 19:55:02.227416 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:55:02.227370    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:55:03.073494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:03.062006    4519 patch_prober.go:29] interesting pod/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:55:03.073494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:03.062079    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 19:55:03.120494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:03.110597    4519 generic.go:356] "Generic (PLEG): container finished" podID="e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" containerID="b409047454d51999e63bf505a6e27edd906a457c829771ab9419a8dd4198e93d" exitCode=0
Oct 30 19:55:03.120494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:03.110948    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy" containerID="cri-o://14d52b7d681427d8cfbee9207df09388eceaaba88b5212ffec8dcd02b78ea0b4" gracePeriod=30
Oct 30 19:55:03.120494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:03.111160    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx" event={"ID":"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b","Type":"ContainerDied","Data":"b409047454d51999e63bf505a6e27edd906a457c829771ab9419a8dd4198e93d"}
Oct 30 19:55:04.626432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.626389    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:04.725156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.725121    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-xks-ca\") pod \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") "
Oct 30 19:55:04.725561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.725432    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-openshift-ca\") pod \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") "
Oct 30 19:55:04.725835 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.725807    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-openshift-ca" pod "e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" (UID: "e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:04.725930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.725871    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-kube-api-access-w9qbk\" (UniqueName: \"kubernetes.io/projected/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-kube-api-access-w9qbk\") pod \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\" (UID: \"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b\") "
Oct 30 19:55:04.727159 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.727135    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:04.727577 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.727548    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-xks-ca" pod "e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" (UID: "e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:04.728381 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.728353    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-kube-api-access-w9qbk" pod "e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" (UID: "e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b"). InnerVolumeSpecName "kube-api-access-w9qbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:04.782230 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.781591    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:55:04.828780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.828742    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:04.828780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:04.828777    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9qbk\" (UniqueName: \"kubernetes.io/projected/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b-kube-api-access-w9qbk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:05.147488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:05.146323    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"
Oct 30 19:55:05.147488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:05.146319    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx" event={"ID":"e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b","Type":"ContainerDied","Data":"1832b04d942684494b9b125dc218a958a3df9d3a2282e848bc85100e5cd09101"}
Oct 30 19:55:05.147488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:05.146399    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1832b04d942684494b9b125dc218a958a3df9d3a2282e848bc85100e5cd09101"
Oct 30 19:55:12.381123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.378384    4519 generic.go:356] "Generic (PLEG): container finished" podID="187df1cd-7922-4595-b165-6489bedda57a" containerID="14d52b7d681427d8cfbee9207df09388eceaaba88b5212ffec8dcd02b78ea0b4" exitCode=0
Oct 30 19:55:12.381123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.378461    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" event={"ID":"187df1cd-7922-4595-b165-6489bedda57a","Type":"ContainerDied","Data":"14d52b7d681427d8cfbee9207df09388eceaaba88b5212ffec8dcd02b78ea0b4"}
Oct 30 19:55:12.381123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.378567    4519 scope.go:122] "RemoveContainer" containerID="05ad786395e636b3061fa5a313d2b437481305808b5bbbb8e17af571bbfd2b9e"
Oct 30 19:55:12.395615 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.395577    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:55:12.439599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439561    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-data\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.439799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439647    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187df1cd-7922-4595-b165-6489bedda57a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187df1cd-7922-4595-b165-6489bedda57a-istiod-ca-cert\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.439799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439723    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-socket\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.439799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439759    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-envoy\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.439799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439786    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/187df1cd-7922-4595-b165-6489bedda57a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187df1cd-7922-4595-b165-6489bedda57a-istio-podinfo\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.440053 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439836    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-istio-token\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-istio-token\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.440053 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439867    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-data" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:12.440053 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.439897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-certs\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.440673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.440389    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-socket" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:12.440673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.440549    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-kube-api-access-lc9b9\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-kube-api-access-lc9b9\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.440673 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.440650    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-credential-socket\") pod \"187df1cd-7922-4595-b165-6489bedda57a\" (UID: \"187df1cd-7922-4595-b165-6489bedda57a\") "
Oct 30 19:55:12.441373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.440547    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-certs" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:12.441373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.440647    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187df1cd-7922-4595-b165-6489bedda57a-istiod-ca-cert" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.443527    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-envoy" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.443541    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/187df1cd-7922-4595-b165-6489bedda57a-istio-podinfo" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.444702    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-istio-token" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.446136    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.446170    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/187df1cd-7922-4595-b165-6489bedda57a-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.446187    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.446201    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.446217    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/187df1cd-7922-4595-b165-6489bedda57a-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.446236    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.446251    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.459207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.449045    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-credential-socket" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:12.461804 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.461753    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-kube-api-access-lc9b9" pod "187df1cd-7922-4595-b165-6489bedda57a" (UID: "187df1cd-7922-4595-b165-6489bedda57a"). InnerVolumeSpecName "kube-api-access-lc9b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:12.548373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.548321    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9b9\" (UniqueName: \"kubernetes.io/projected/187df1cd-7922-4595-b165-6489bedda57a-kube-api-access-lc9b9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:12.548373 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:12.548366    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/187df1cd-7922-4595-b165-6489bedda57a-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:55:13.396084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:13.396050    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"
Oct 30 19:55:13.396084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:13.396073    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4" event={"ID":"187df1cd-7922-4595-b165-6489bedda57a","Type":"ContainerDied","Data":"182b113cc7c740d8390dfc35b0ad3da41eb14b8743a9abf264ee4b8e3ceaffb1"}
Oct 30 19:55:13.396732 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:13.396149    4519 scope.go:122] "RemoveContainer" containerID="14d52b7d681427d8cfbee9207df09388eceaaba88b5212ffec8dcd02b78ea0b4"
Oct 30 19:55:13.494493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:13.494436    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"]
Oct 30 19:55:13.500801 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:13.500759    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-75082b11-gw-openshift-default-7ccb8cbbb5-fbqw4"]
Oct 30 19:55:13.615236 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:13.615201    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187df1cd-7922-4595-b165-6489bedda57a" path="/var/lib/kubelet/pods/187df1cd-7922-4595-b165-6489bedda57a/volumes"
Oct 30 19:55:32.228867 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:55:32.228813    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:55:33.117989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.117948    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"]
Oct 30 19:55:33.119291 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119185    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" containerName="revoke-keys"
Oct 30 19:55:33.119291 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119217    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" containerName="revoke-keys"
Oct 30 19:55:33.119291 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119281    4519 cpu_manager.go:414] "containerMap: removing container" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy"
Oct 30 19:55:33.119291 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119290    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy"
Oct 30 19:55:33.119717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119308    4519 cpu_manager.go:414] "containerMap: removing container" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy"
Oct 30 19:55:33.119717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119317    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy"
Oct 30 19:55:33.119717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119499    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy"
Oct 30 19:55:33.119717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119515    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="187df1cd-7922-4595-b165-6489bedda57a" containerName="istio-proxy"
Oct 30 19:55:33.119717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.119534    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" containerName="revoke-keys"
Oct 30 19:55:33.123879 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.123853    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.143142 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.142927    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c8fa69-openshift-default-dockercfg-hvqzn\""
Oct 30 19:55:33.173532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.171882    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"]
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.211766    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-data\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.211837    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-istio-token\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.211887    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb4412e-9702-4277-bb1c-3f658f977837-istio-podinfo\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.211962    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-envoy\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.212049    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8hh\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-kube-api-access-ql8hh\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.212109    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-certs\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.212136    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-credential-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.212233    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.213823 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.212271    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb4412e-9702-4277-bb1c-3f658f977837-istiod-ca-cert\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.325819    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-data\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.325894    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-istio-token\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.325951    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb4412e-9702-4277-bb1c-3f658f977837-istio-podinfo\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.326030    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-envoy\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.326123    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8hh\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-kube-api-access-ql8hh\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.326188    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-certs\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.326215    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-credential-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.326317    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.326358    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb4412e-9702-4277-bb1c-3f658f977837-istiod-ca-cert\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.329508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.329260    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-data\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.330505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.330001    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-certs\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.334492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.330564    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-credential-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.334492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.330835    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-socket\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.364502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.362324    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb4412e-9702-4277-bb1c-3f658f977837-istiod-ca-cert\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.365869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.365805    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb4412e-9702-4277-bb1c-3f658f977837-istio-podinfo\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.397987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.396797    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8hh\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-kube-api-access-ql8hh\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.397987 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.397858    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-envoy\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.419327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.419253    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-istio-token\") pod \"e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") " pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:33.460491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:33.459814    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:34.061458 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:34.061348    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"]
Oct 30 19:55:34.083256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:34.083205    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:34.083462 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:34.083308    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:34.083462 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:34.083371    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:34.882927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:34.882890    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" event={"ID":"ccb4412e-9702-4277-bb1c-3f658f977837","Type":"ContainerStarted","Data":"817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f"}
Oct 30 19:55:34.883591 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:34.882947    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" event={"ID":"ccb4412e-9702-4277-bb1c-3f658f977837","Type":"ContainerStarted","Data":"0796c5930844092fcfacad76ab25b19a8a97ecf8e0c364f84e8a952463c6d626"}
Oct 30 19:55:34.950940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:34.950870    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podStartSLOduration=1.950850205 podStartE2EDuration="1.950850205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:34.938670084 +0000 UTC m=+1259.439373810" watchObservedRunningTime="2026-10-30 19:55:34.950850205 +0000 UTC m=+1259.451553931"
Oct 30 19:55:35.467995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:35.466127    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:35.484507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:35.484428    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:35.896021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:35.895929    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:35.897064 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:35.897042    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 19:55:39.294910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.294878    4519 scope.go:122] "RemoveContainer" containerID="beda752664e5aed32b301add03c37ed2c786919e02ed2fd0c9511736e439ed15"
Oct 30 19:55:39.312352 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.312054    4519 scope.go:122] "RemoveContainer" containerID="eacf0d5c83202b22eb46124b6f53fe95f89ac9e03ea67c91a816e86a86dd25e2"
Oct 30 19:55:39.328138 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.328107    4519 scope.go:122] "RemoveContainer" containerID="14bcd90bb90371413dbae5de8b3e377c0d26a0387369639c1c487e362b2fc8df"
Oct 30 19:55:39.356179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.356145    4519 scope.go:122] "RemoveContainer" containerID="ea7e8f2b4d915d164a91d77a1d9d9368999f3fb4b50600c12ca07810758a262c"
Oct 30 19:55:39.402181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.387145    4519 scope.go:122] "RemoveContainer" containerID="add205026179aa0adafd8d3f8ed9719dd810b847cab93ff2975674908422c6af"
Oct 30 19:55:39.650717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.650687    4519 scope.go:122] "RemoveContainer" containerID="0e7679040cc801bc097d34b8a8bf011d0918b68b05fb00a03693fc2f36e8b639"
Oct 30 19:55:39.666986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.666958    4519 scope.go:122] "RemoveContainer" containerID="41104fbaae2c42b017bda3db037ff24b39efd9c2114fc63728e9da187e87db46"
Oct 30 19:55:39.680998 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.680974    4519 scope.go:122] "RemoveContainer" containerID="1b5375b5071d4f72e7ec20fc4a0e74557ebebab3718958030b855872ecdfd174"
Oct 30 19:55:39.693437 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.693401    4519 scope.go:122] "RemoveContainer" containerID="08fd9f01ae55d49fe6ab2417605148545081787dab1b95e89383114b2e1a6958"
Oct 30 19:55:39.710537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.710508    4519 scope.go:122] "RemoveContainer" containerID="22d95e2369b05a4eae1b67d501d0b62e0bf0d58769a858a4997ced09b6b3cec1"
Oct 30 19:55:39.727579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.727438    4519 scope.go:122] "RemoveContainer" containerID="5fc9fb86bca25d6131ad630f3ab56d1e7abbfa9a944dcc963f177901a4881d12"
Oct 30 19:55:39.765394 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.765072    4519 scope.go:122] "RemoveContainer" containerID="79c307fc4aadb09acbf98d1bbf7940c0316ab5f7da1063f1fbef7e7c464e56d4"
Oct 30 19:55:39.777899 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.777871    4519 scope.go:122] "RemoveContainer" containerID="675bebcbad23673af6c52df4e4757d348e0317d6561a4f50d83cec8d47dee6ee"
Oct 30 19:55:39.790113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.790082    4519 scope.go:122] "RemoveContainer" containerID="60633fb7f7fb846b2ff9ca8f26d0614c32691ad0701e0cce21590f9162262f40"
Oct 30 19:55:39.801698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.801662    4519 scope.go:122] "RemoveContainer" containerID="1b980bf8e2e72064b594d97901450caa42a3e8a7b26e944eef2543acada3f9f7"
Oct 30 19:55:39.813331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.813308    4519 scope.go:122] "RemoveContainer" containerID="cf05c25ba45645c7114ca1f46dfb350da7c3647d6200672d24eb7b359ac8f3a4"
Oct 30 19:55:39.825837 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:39.825816    4519 scope.go:122] "RemoveContainer" containerID="9d74fe23b15b630116ae444769a01a02a990a8d9a2836c0bee06ec36a0b4746d"
Oct 30 19:55:42.926125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:42.926082    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"]
Oct 30 19:55:42.936615 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:42.935773    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:42.978492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:42.963332    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"]
Oct 30 19:55:42.978492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:42.971883    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-159b3a-openshift-default-dockercfg-8tw4f\""
Oct 30 19:55:43.014860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.014812    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-token\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.015258 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.015226    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-podinfo\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.015572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.015537    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-certs\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.015688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.015643    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-credential-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.015975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.015825    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-envoy\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.015975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.015860    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istiod-ca-cert\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.016124 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.016014    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjw5\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-kube-api-access-wrjw5\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.016124 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.016091    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-data\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.016235 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.016138    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.144211 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.144171    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjw5\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-kube-api-access-wrjw5\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.144554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.144517    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-data\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.144698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.144574    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.144698 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.144622    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-token\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.144817 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.144730    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-podinfo\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.144947 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.144929    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-certs\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.145017 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.144967    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-credential-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.145076 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.145047    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-envoy\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.145136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.145075    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istiod-ca-cert\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.146094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.146064    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istiod-ca-cert\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.162494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.151200    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-data\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.163375 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.157194    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-certs\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.163596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.157414    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-credential-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.163737 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.158094    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.169601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.166875    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-envoy\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.192355 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.192265    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjw5\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-kube-api-access-wrjw5\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.202137 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.202101    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-token\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.214489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.209562    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-podinfo\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.300130 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.299802    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:43.811571 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:55:43.811536    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39e72f2_f9c1_4d6a_b581_99d1a2555536.slice/crio-51e5ec7b9357660501c420f4384963c23ff7fee9e2de614a939dca4bb943e71b WatchSource:0}: Error finding container 51e5ec7b9357660501c420f4384963c23ff7fee9e2de614a939dca4bb943e71b: Status 404 returned error can't find the container with id 51e5ec7b9357660501c420f4384963c23ff7fee9e2de614a939dca4bb943e71b
Oct 30 19:55:43.814676 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.814633    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:43.814813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.814731    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:43.814813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.814772    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:43.816981 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:43.816949    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"]
Oct 30 19:55:44.023984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:44.023088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" event={"ID":"e39e72f2-f9c1-4d6a-b581-99d1a2555536","Type":"ContainerStarted","Data":"ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac"}
Oct 30 19:55:44.023984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:44.023163    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" event={"ID":"e39e72f2-f9c1-4d6a-b581-99d1a2555536","Type":"ContainerStarted","Data":"51e5ec7b9357660501c420f4384963c23ff7fee9e2de614a939dca4bb943e71b"}
Oct 30 19:55:44.301434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:44.301394    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:44.304371 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:44.304331    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:55:44.304590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:44.304421    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:55:45.305855 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:45.305820    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:45.375906 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:45.375830    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podStartSLOduration=3.375808469 podStartE2EDuration="3.375808469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:44.076892227 +0000 UTC m=+1268.577595952" watchObservedRunningTime="2026-10-30 19:55:45.375808469 +0000 UTC m=+1269.876512194"
Oct 30 19:55:46.047302 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:46.047269    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:46.048233 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:46.048211    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:55:58.036679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:58.036636    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"]
Oct 30 19:55:58.048827 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:58.048790    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa61a8da-tjpp6"]
Oct 30 19:55:59.615160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:55:59.615120    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c53bd7-b699-4371-b434-d7882000083a" path="/var/lib/kubelet/pods/b1c53bd7-b699-4371-b434-d7882000083a/volumes"
Oct 30 19:56:00.037746 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:00.037693    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"]
Oct 30 19:56:00.045453 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:00.045403    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e21050a-l7crz"]
Oct 30 19:56:01.617160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:01.617125    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f99933f-b038-49a6-a443-166cd10e00ea" path="/var/lib/kubelet/pods/4f99933f-b038-49a6-a443-166cd10e00ea/volumes"
Oct 30 19:56:02.226273 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:56:02.226233    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:56:09.261910 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.261861    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"]
Oct 30 19:56:09.315021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.314691    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"]
Oct 30 19:56:09.315021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.314818    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.331268 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.331181    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2224e496-openshift-default-dockercfg-xxkmv\""
Oct 30 19:56:09.430362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430319    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba3d21-8105-42b2-bc93-abfb6953efca-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.430590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430394    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.430590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430533    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.430684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430608    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.430684 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430667    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.430790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430775    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.430952 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430934    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmzm\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-kube-api-access-bdmzm\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.431008 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.430990    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.431171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.431153    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.534899 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.534797    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba3d21-8105-42b2-bc93-abfb6953efca-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.535225 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.535202    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.535385 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.535365    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.535563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.535546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.535706 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.535690    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.535886 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.535871    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.536143 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.536126    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmzm\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-kube-api-access-bdmzm\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.536291 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.536277    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.536550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.536528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.553494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.538023    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba3d21-8105-42b2-bc93-abfb6953efca-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.554213 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.554178    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.554661 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.554638    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.555395 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.555372    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.556354 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.556329    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.564493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.558240    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.571255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.571212    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.589493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.589349    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmzm\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-kube-api-access-bdmzm\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.609227 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.609179    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:09.659488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:09.645494    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:10.160215 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:56:10.160181    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ba3d21_8105_42b2_bc93_abfb6953efca.slice/crio-39972511aeefcb187ef0f859194f41dd2fe4d51f8718500e0e5b4b060fc05073 WatchSource:0}: Error finding container 39972511aeefcb187ef0f859194f41dd2fe4d51f8718500e0e5b4b060fc05073: Status 404 returned error can't find the container with id 39972511aeefcb187ef0f859194f41dd2fe4d51f8718500e0e5b4b060fc05073
Oct 30 19:56:10.161776 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.161749    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"]
Oct 30 19:56:10.163808 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.163771    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:10.163944 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.163924    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:10.164009 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.163976    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:10.394765 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.394724    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" event={"ID":"40ba3d21-8105-42b2-bc93-abfb6953efca","Type":"ContainerStarted","Data":"2256ce950e5c68549704876d5f94a28c3b7c26a010c3899121a4d32118a3b60f"}
Oct 30 19:56:10.395306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.394823    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" event={"ID":"40ba3d21-8105-42b2-bc93-abfb6953efca","Type":"ContainerStarted","Data":"39972511aeefcb187ef0f859194f41dd2fe4d51f8718500e0e5b4b060fc05073"}
Oct 30 19:56:10.447990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.447914    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podStartSLOduration=1.447896441 podStartE2EDuration="1.447896441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:10.440825565 +0000 UTC m=+1294.941529291" watchObservedRunningTime="2026-10-30 19:56:10.447896441 +0000 UTC m=+1294.948600164"
Oct 30 19:56:10.659489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:10.650647    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:11.657339 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:11.657293    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:12.416961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:12.416924    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:12.418140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:12.418114    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:56:28.052875 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:28.052783    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"]
Oct 30 19:56:28.064956 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:28.064907    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1095500b-qtjr5"]
Oct 30 19:56:29.617893 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:29.617843    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3472ce-3ec5-4c49-adca-cea04950ebb1" path="/var/lib/kubelet/pods/1c3472ce-3ec5-4c49-adca-cea04950ebb1/volumes"
Oct 30 19:56:30.043179 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:30.043136    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"]
Oct 30 19:56:30.056391 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:30.056351    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425381a4-s6z98"]
Oct 30 19:56:31.614361 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:31.614326    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192871e4-b312-4780-95b5-20aa6ef86770" path="/var/lib/kubelet/pods/192871e4-b312-4780-95b5-20aa6ef86770/volumes"
Oct 30 19:56:32.227485 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:56:32.227435    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:56:36.038536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:36.038486    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"]
Oct 30 19:56:36.082771 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:36.082725    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52db9798-bhtf9"]
Oct 30 19:56:37.621139 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:37.621091    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48745e1-df7e-4d2e-8c16-29a22f25d533" path="/var/lib/kubelet/pods/a48745e1-df7e-4d2e-8c16-29a22f25d533/volumes"
Oct 30 19:56:39.984978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:39.984934    4519 scope.go:122] "RemoveContainer" containerID="338c090ce2b767ea829458aec5cbc0bb90ab1ad7d1402ac7553629766dcbc01a"
Oct 30 19:56:41.313453 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.313415    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"]
Oct 30 19:56:41.318199 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.318174    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.340096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.340057    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"]
Oct 30 19:56:41.342940 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.342898    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:41.343384 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.343364    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:56:41.456688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.456649    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg2m4\" (UniqueName: \"kubernetes.io/projected/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-kube-api-access-bg2m4\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.456909 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.456861    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.457015 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.456984    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.562249 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.562153    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg2m4\" (UniqueName: \"kubernetes.io/projected/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-kube-api-access-bg2m4\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.562832 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.562797    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.563084 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.563065    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.564327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.564214    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.567372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.567287    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.621801 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.612113    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg2m4\" (UniqueName: \"kubernetes.io/projected/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-kube-api-access-bg2m4\") pod \"maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.639330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.639282    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:41.987322 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:56:41.987283    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d792d80_01e6_43be_84a6_e78a6cb0eb8e.slice/crio-42c7a3fa30d502c4baad626e33264c273476c67668bd3028e803f483a6c4ab9d WatchSource:0}: Error finding container 42c7a3fa30d502c4baad626e33264c273476c67668bd3028e803f483a6c4ab9d: Status 404 returned error can't find the container with id 42c7a3fa30d502c4baad626e33264c273476c67668bd3028e803f483a6c4ab9d
Oct 30 19:56:41.990700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:41.990614    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"]
Oct 30 19:56:42.034232 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.034191    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"]
Oct 30 19:56:42.039931 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.039894    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0fa03f8-pf6nz"]
Oct 30 19:56:42.133306 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.133269    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l" event={"ID":"0d792d80-01e6-43be-84a6-e78a6cb0eb8e","Type":"ContainerStarted","Data":"42c7a3fa30d502c4baad626e33264c273476c67668bd3028e803f483a6c4ab9d"}
Oct 30 19:56:42.135529 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.135493    4519 generic.go:356] "Generic (PLEG): container finished" podID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerID="ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac" exitCode=0
Oct 30 19:56:42.135705 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.135592    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" event={"ID":"e39e72f2-f9c1-4d6a-b581-99d1a2555536","Type":"ContainerDied","Data":"ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac"}
Oct 30 19:56:42.136181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.136154    4519 scope.go:122] "RemoveContainer" containerID="ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac"
Oct 30 19:56:42.142411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.140811    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:42.142411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.140929    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:42.142411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:42.140986    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:43.031228 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.031186    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"]
Oct 30 19:56:43.043508 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.043452    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b467df3-trq2r"]
Oct 30 19:56:43.153544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.149026    4519 generic.go:356] "Generic (PLEG): container finished" podID="0d792d80-01e6-43be-84a6-e78a6cb0eb8e" containerID="6026b2c6bcbe73ccbbf6f27a9578b4cddacad7fe93eb5da24a20fab39c2feddd" exitCode=0
Oct 30 19:56:43.153544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.149072    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l" event={"ID":"0d792d80-01e6-43be-84a6-e78a6cb0eb8e","Type":"ContainerDied","Data":"6026b2c6bcbe73ccbbf6f27a9578b4cddacad7fe93eb5da24a20fab39c2feddd"}
Oct 30 19:56:43.153544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.151804    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" event={"ID":"e39e72f2-f9c1-4d6a-b581-99d1a2555536","Type":"ContainerStarted","Data":"64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b"}
Oct 30 19:56:43.301190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.301063    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:56:43.302884 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.302845    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:56:43.303098 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.302921    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:56:43.633436 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.633334    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27956871-d80b-4a5b-bfd5-3bbd0d5b6968" path="/var/lib/kubelet/pods/27956871-d80b-4a5b-bfd5-3bbd0d5b6968/volumes"
Oct 30 19:56:43.634314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:43.634282    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518e2010-cfb2-4726-a42f-4f0cfb7ff3a9" path="/var/lib/kubelet/pods/518e2010-cfb2-4726-a42f-4f0cfb7ff3a9/volumes"
Oct 30 19:56:44.086125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.086052    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"]
Oct 30 19:56:44.093813 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.093777    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adc4e00-48wx9"]
Oct 30 19:56:44.301500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.300989    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:56:44.301500 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.301058    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:56:44.442576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.442540    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:44.525894 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.524304    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-xks-ca\") pod \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") "
Oct 30 19:56:44.525894 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.524404    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-kube-api-access-bg2m4\" (UniqueName: \"kubernetes.io/projected/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-kube-api-access-bg2m4\") pod \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") "
Oct 30 19:56:44.525894 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.524661    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-openshift-ca\") pod \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\" (UID: \"0d792d80-01e6-43be-84a6-e78a6cb0eb8e\") "
Oct 30 19:56:44.525894 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.525179    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-openshift-ca" pod "0d792d80-01e6-43be-84a6-e78a6cb0eb8e" (UID: "0d792d80-01e6-43be-84a6-e78a6cb0eb8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:44.526262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.526039    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:56:44.529013 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.528975    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-xks-ca" pod "0d792d80-01e6-43be-84a6-e78a6cb0eb8e" (UID: "0d792d80-01e6-43be-84a6-e78a6cb0eb8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:44.529182 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.529015    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-kube-api-access-bg2m4" pod "0d792d80-01e6-43be-84a6-e78a6cb0eb8e" (UID: "0d792d80-01e6-43be-84a6-e78a6cb0eb8e"). InnerVolumeSpecName "kube-api-access-bg2m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:44.628066 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.628028    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:56:44.628066 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:44.628062    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg2m4\" (UniqueName: \"kubernetes.io/projected/0d792d80-01e6-43be-84a6-e78a6cb0eb8e-kube-api-access-bg2m4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:56:45.034003 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.033960    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"]
Oct 30 19:56:45.039572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.039535    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b847c1ab-dhjnw"]
Oct 30 19:56:45.204512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.204448    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"
Oct 30 19:56:45.204925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.204447    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l" event={"ID":"0d792d80-01e6-43be-84a6-e78a6cb0eb8e","Type":"ContainerDied","Data":"42c7a3fa30d502c4baad626e33264c273476c67668bd3028e803f483a6c4ab9d"}
Oct 30 19:56:45.204925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.204611    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42c7a3fa30d502c4baad626e33264c273476c67668bd3028e803f483a6c4ab9d"
Oct 30 19:56:45.301760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.301660    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:56:45.301760 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.301725    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:56:45.620914 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.620829    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f383ca-df7f-4185-8aac-d6f3b12dd2c2" path="/var/lib/kubelet/pods/02f383ca-df7f-4185-8aac-d6f3b12dd2c2/volumes"
Oct 30 19:56:45.621658 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:45.621634    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05df8544-fb87-4486-baa5-e2d107f355ee" path="/var/lib/kubelet/pods/05df8544-fb87-4486-baa5-e2d107f355ee/volumes"
Oct 30 19:56:46.053652 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.053615    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:56:46.254490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.249531    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"]
Oct 30 19:56:46.254490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.250669    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0d792d80-01e6-43be-84a6-e78a6cb0eb8e" containerName="revoke-keys"
Oct 30 19:56:46.254490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.250693    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d792d80-01e6-43be-84a6-e78a6cb0eb8e" containerName="revoke-keys"
Oct 30 19:56:46.254490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.251026    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d792d80-01e6-43be-84a6-e78a6cb0eb8e" containerName="revoke-keys"
Oct 30 19:56:46.302493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.295880    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.328545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.309217    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:56:46.328545 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.309342    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:56:46.329183 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.329151    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8d97a5-openshift-default-dockercfg-s448f\""
Oct 30 19:56:46.332252 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.332219    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"]
Oct 30 19:56:46.524491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.502858    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-envoy\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.537492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.527604    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc2d597-f684-4579-ba47-7735c112706a-istio-podinfo\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.569610 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.537895    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc2d597-f684-4579-ba47-7735c112706a-istiod-ca-cert\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.570044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.570013    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-socket\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.599562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.582648    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-istio-token\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.612408 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.599860    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-certs\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.626664 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.626621    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbqf\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-kube-api-access-5tbqf\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.639714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.639665    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-data\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.652521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.639985    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-credential-socket\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.809797 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.809762    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-data\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.810205 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.810180    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-credential-socket\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.810794 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.810773    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-envoy\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.811163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.811132    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc2d597-f684-4579-ba47-7735c112706a-istio-podinfo\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.812218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.812194    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc2d597-f684-4579-ba47-7735c112706a-istiod-ca-cert\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.812772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.812751    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-socket\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.814035 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.812003    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-data\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.814208 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.813282    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-socket\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.814345 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.813608    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-credential-socket\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.814595 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.814577    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-istio-token\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.814781 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.814755    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-certs\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.814989 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.814972    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbqf\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-kube-api-access-5tbqf\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.817049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.817029    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-certs\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.817371 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.817352    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc2d597-f684-4579-ba47-7735c112706a-istiod-ca-cert\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.829100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.828985    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc2d597-f684-4579-ba47-7735c112706a-istio-podinfo\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.843983 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.840992    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-envoy\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.844190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.844113    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-istio-token\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.866550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.849209    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbqf\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-kube-api-access-5tbqf\") pod \"e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") " pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:46.927412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:46.927366    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:47.320662 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:47.320622    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:56:47.327631 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:47.327567    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:56:47.965850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:47.964568    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"]
Oct 30 19:56:48.025912 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.025864    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:48.026239 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.026215    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:48.026414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.026373    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:48.302635 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.301021    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:56:48.302635 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.301105    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:56:48.433190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.431862    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" event={"ID":"2fc2d597-f684-4579-ba47-7735c112706a","Type":"ContainerStarted","Data":"76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977"}
Oct 30 19:56:48.433190 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.431926    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" event={"ID":"2fc2d597-f684-4579-ba47-7735c112706a","Type":"ContainerStarted","Data":"0482dba3fdd1ce981a9e784479cc952bf80e640589c56985a30c6235ada75bf6"}
Oct 30 19:56:48.501493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.496989    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podStartSLOduration=2.496963112 podStartE2EDuration="2.496963112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:48.481943662 +0000 UTC m=+1332.982647387" watchObservedRunningTime="2026-10-30 19:56:48.496963112 +0000 UTC m=+1332.997666836"
Oct 30 19:56:48.927990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:48.927941    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:49.038262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:49.038225    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"]
Oct 30 19:56:49.059491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:49.056301    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ce79df-drk68"]
Oct 30 19:56:49.301052 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:49.300941    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:56:49.301052 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:49.301001    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:56:49.616954 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:49.616861    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0619998a-5e47-4086-8986-728b863e3827" path="/var/lib/kubelet/pods/0619998a-5e47-4086-8986-728b863e3827/volumes"
Oct 30 19:56:49.934850 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:49.934821    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:50.044286 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.044245    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"]
Oct 30 19:56:50.052269 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.052192    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e33d43-ctl4c"]
Oct 30 19:56:50.306160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.306072    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:56:50.307530 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.307502    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 19:56:50.340282 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.340241    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"]
Oct 30 19:56:50.348601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.348563    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.367542 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.367005    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-93263c-openshift-default-dockercfg-9qxbr\""
Oct 30 19:56:50.381991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.381799    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"]
Oct 30 19:56:50.478417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.478055    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:50.479433 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.479365    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:56:50.514038 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.513999    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-data\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514049    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-credential-socket\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514150    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgfkm\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-kube-api-access-wgfkm\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514355 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514224    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-podinfo\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514419 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514400    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ba69a-0c7c-45da-9e5d-16b26139248b-istiod-ca-cert\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514584    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-socket\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514624    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-envoy\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514689    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-token\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.514772 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.514757    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-certs\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.617373    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ba69a-0c7c-45da-9e5d-16b26139248b-istiod-ca-cert\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.617528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-socket\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.617554    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-envoy\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.617606    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-token\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.617652    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-certs\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.617912    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-data\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.617977    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-credential-socket\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.618042    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgfkm\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-kube-api-access-wgfkm\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.618075    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-podinfo\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.618348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.618206    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ba69a-0c7c-45da-9e5d-16b26139248b-istiod-ca-cert\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.619868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.619819    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-data\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.619868 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.619839    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-socket\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.620049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.619913    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-certs\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.620171 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.620148    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-credential-socket\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.622350 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.622320    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-podinfo\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.622614 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.622591    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-envoy\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.638532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.638447    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgfkm\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-kube-api-access-wgfkm\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.638727 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.638457    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-token\") pod \"e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") " pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:50.699279 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:50.697992    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:51.436325 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:56:51.436295    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00ba69a_0c7c_45da_9e5d_16b26139248b.slice/crio-3ecfcbf084970918d79322cafa4a4bf6d776ccab5f968f7b1b56fac86d85bedc WatchSource:0}: Error finding container 3ecfcbf084970918d79322cafa4a4bf6d776ccab5f968f7b1b56fac86d85bedc: Status 404 returned error can't find the container with id 3ecfcbf084970918d79322cafa4a4bf6d776ccab5f968f7b1b56fac86d85bedc
Oct 30 19:56:51.440481 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:51.440423    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:51.440593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:51.440540    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:51.440593 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:51.440585    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:56:51.440836 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:51.440809    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"]
Oct 30 19:56:51.491643 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:51.491595    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" event={"ID":"d00ba69a-0c7c-45da-9e5d-16b26139248b","Type":"ContainerStarted","Data":"3ecfcbf084970918d79322cafa4a4bf6d776ccab5f968f7b1b56fac86d85bedc"}
Oct 30 19:56:51.830596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:51.800234    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1f4334-51b3-4dfe-b468-3f09846ba9db" path="/var/lib/kubelet/pods/2a1f4334-51b3-4dfe-b468-3f09846ba9db/volumes"
Oct 30 19:56:52.549977 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:52.549881    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" event={"ID":"d00ba69a-0c7c-45da-9e5d-16b26139248b","Type":"ContainerStarted","Data":"f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0"}
Oct 30 19:56:52.611201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:52.611132    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podStartSLOduration=2.6110690720000003 podStartE2EDuration="2.611069072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:52.606911456 +0000 UTC m=+1337.107615183" watchObservedRunningTime="2026-10-30 19:56:52.611069072 +0000 UTC m=+1337.111772797"
Oct 30 19:56:52.722177 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:52.722142    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:52.736513 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:52.735770    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:53.565669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:53.565636    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:56:53.567952 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:56:53.567905    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:57:02.222023 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:57:02.221976    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:57:04.039152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:04.039114    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"]
Oct 30 19:57:04.053131 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:04.053096    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09142759-wqkdw"]
Oct 30 19:57:05.032359 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:05.032321    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"]
Oct 30 19:57:05.044290 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:05.044248    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e77105-f6mvp"]
Oct 30 19:57:05.619669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:05.619633    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f950276-7c06-4ba2-bf00-3523d9e1002a" path="/var/lib/kubelet/pods/2f950276-7c06-4ba2-bf00-3523d9e1002a/volumes"
Oct 30 19:57:05.620272 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:05.620253    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da36518-96eb-400f-9cbf-370aad08ea8e" path="/var/lib/kubelet/pods/3da36518-96eb-400f-9cbf-370aad08ea8e/volumes"
Oct 30 19:57:09.361538 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:09.359390    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"]
Oct 30 19:57:09.371410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:09.371363    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c53b7887-m2x7l"]
Oct 30 19:57:09.619051 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:09.618898    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d792d80-01e6-43be-84a6-e78a6cb0eb8e" path="/var/lib/kubelet/pods/0d792d80-01e6-43be-84a6-e78a6cb0eb8e/volumes"
Oct 30 19:57:13.035278 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.035241    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"]
Oct 30 19:57:13.046143 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.046113    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa470dd-c74bb"]
Oct 30 19:57:13.429372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.429327    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"]
Oct 30 19:57:13.429664 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.429633    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy" containerID="cri-o://14c935898a5f231cda9d90fb94c96539039b10dc3380e6b26342512a85e00bdb" gracePeriod=30
Oct 30 19:57:13.442114 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.442073    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"]
Oct 30 19:57:13.442403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.442372    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy" containerID="cri-o://b194a079bc053415ea0f76ac9b92d86a965d9c85dc71dde808b3954d2d6627b4" gracePeriod=30
Oct 30 19:57:13.454018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.453974    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"]
Oct 30 19:57:13.454659 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.454567    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy" containerID="cri-o://717cac305f3d8aa180c449d9f23564b1fcda0013912d27c8fb399a14f56d7550" gracePeriod=30
Oct 30 19:57:13.623128 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:13.622845    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee27653-d254-4603-baee-c5e24346fc72" path="/var/lib/kubelet/pods/9ee27653-d254-4603-baee-c5e24346fc72/volumes"
Oct 30 19:57:19.317725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.317685    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"]
Oct 30 19:57:19.450229 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.450146    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"]
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.549664    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-token\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.549777    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-podinfo\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.549860    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-envoy\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.549899    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-data\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.550047    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-credential-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.550098    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.550171    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wb8z\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-kube-api-access-9wb8z\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.550284    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-certs\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.556486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.550337    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istiod-ca-cert\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.568965 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.566442    4519 generic.go:356] "Generic (PLEG): container finished" podID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerID="14c935898a5f231cda9d90fb94c96539039b10dc3380e6b26342512a85e00bdb" exitCode=0
Oct 30 19:57:19.570700 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.570654    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" event={"ID":"591e7d80-491c-4c58-802a-fc833b7ea25c","Type":"ContainerDied","Data":"14c935898a5f231cda9d90fb94c96539039b10dc3380e6b26342512a85e00bdb"}
Oct 30 19:57:19.588499 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.588441    4519 generic.go:356] "Generic (PLEG): container finished" podID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerID="b194a079bc053415ea0f76ac9b92d86a965d9c85dc71dde808b3954d2d6627b4" exitCode=0
Oct 30 19:57:19.588967 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.588939    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" event={"ID":"be62cbee-3784-4ec7-905c-f2751cf55f0e","Type":"ContainerDied","Data":"b194a079bc053415ea0f76ac9b92d86a965d9c85dc71dde808b3954d2d6627b4"}
Oct 30 19:57:19.594416 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.591975    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.612097 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.609780    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a165e5-openshift-default-dockercfg-wc9pt\""
Oct 30 19:57:19.618204 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.618127    4519 generic.go:356] "Generic (PLEG): container finished" podID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerID="717cac305f3d8aa180c449d9f23564b1fcda0013912d27c8fb399a14f56d7550" exitCode=0
Oct 30 19:57:19.664799 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.664762    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-credential-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.676599 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.665733    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-credential-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.704969 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.704796    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.704969 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.704900    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wb8z\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-kube-api-access-9wb8z\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.705211 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.705013    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-certs\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.705211 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.705064    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istiod-ca-cert\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.706499 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.706204    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-socket\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.707687 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.706921    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-certs\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.707687 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.707600    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istiod-ca-cert\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.708878 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.705172    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-token\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.709150 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.709132    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-podinfo\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.709342 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.709313    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-envoy\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.709489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.709457    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-data\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.710047 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.710025    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-data\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.712758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.712709    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-envoy\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.720792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.720762    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" event={"ID":"4951a800-4431-4b8a-b6b7-2b400d641c2f","Type":"ContainerDied","Data":"717cac305f3d8aa180c449d9f23564b1fcda0013912d27c8fb399a14f56d7550"}
Oct 30 19:57:19.723178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.721870    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-podinfo\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.726412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.726382    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wb8z\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-kube-api-access-9wb8z\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.727789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.727762    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-token\") pod \"e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") " pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:19.775444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.775410    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"]
Oct 30 19:57:19.782997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.782971    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.803150 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.802916    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-27a4d3-openshift-default-dockercfg-qtk88\""
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.812894    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-certs\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.812978    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-credential-socket\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.813009    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ndj\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-kube-api-access-g6ndj\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.813095    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-istio-token\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.813152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-socket\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.813222    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-envoy\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.813300    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6881d466-0536-4777-8072-0dc9d3733166-istiod-ca-cert\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.813448    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6881d466-0536-4777-8072-0dc9d3733166-istio-podinfo\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.821156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.813569    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-data\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:19.832992 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:19.831317    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"]
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.115552    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-certs\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.115629    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-credential-socket\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.115666    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ndj\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-kube-api-access-g6ndj\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.115728    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-istio-token\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.115778    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-socket\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.115841    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-envoy\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.115894    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6881d466-0536-4777-8072-0dc9d3733166-istiod-ca-cert\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.116001    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6881d466-0536-4777-8072-0dc9d3733166-istio-podinfo\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.116065    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-data\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.116779    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-data\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.116832    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:20.117488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.117092    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-socket\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.118238 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.118047    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6881d466-0536-4777-8072-0dc9d3733166-istiod-ca-cert\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.154536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.119677    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-certs\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.154536 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.119923    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-credential-socket\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.192234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.192140    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6881d466-0536-4777-8072-0dc9d3733166-istio-podinfo\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.392413 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.392277    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ndj\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-kube-api-access-g6ndj\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.394921 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.394849    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-envoy\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.409146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.403835    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-istio-token\") pod \"e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") " pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.409146 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.405858    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:57:20.413399 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.413243    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:57:20.430255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.427833    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489006    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-data\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489060    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-data\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489108    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-podinfo\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489151    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-kube-api-access-w9pgt\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-kube-api-access-w9pgt\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489187    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591e7d80-491c-4c58-802a-fc833b7ea25c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591e7d80-491c-4c58-802a-fc833b7ea25c-istiod-ca-cert\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489255    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-envoy\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489290    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be62cbee-3784-4ec7-905c-f2751cf55f0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be62cbee-3784-4ec7-905c-f2751cf55f0e-istiod-ca-cert\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489359    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-data\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489392    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-certs\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489430    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-token\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-token\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.489488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489503    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-envoy\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489538    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-kube-api-access-z69vq\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-kube-api-access-z69vq\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489600    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-credential-socket\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489651    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-token\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-token\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489681    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-podinfo\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489727    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-kube-api-access-jl7xj\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-kube-api-access-jl7xj\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489759    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4951a800-4431-4b8a-b6b7-2b400d641c2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951a800-4431-4b8a-b6b7-2b400d641c2f-istiod-ca-cert\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489795    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-credential-socket\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489842    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-socket\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489882    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-certs\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.489957    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-token\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-token\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.490037    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-podinfo\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.490215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.490210    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-socket\") pod \"be62cbee-3784-4ec7-905c-f2751cf55f0e\" (UID: \"be62cbee-3784-4ec7-905c-f2751cf55f0e\") "
Oct 30 19:57:20.490818 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.490273    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-socket\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.490818 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.490301    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-envoy\") pod \"4951a800-4431-4b8a-b6b7-2b400d641c2f\" (UID: \"4951a800-4431-4b8a-b6b7-2b400d641c2f\") "
Oct 30 19:57:20.490818 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.490382    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-certs\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.490818 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.490478    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-credential-socket\") pod \"591e7d80-491c-4c58-802a-fc833b7ea25c\" (UID: \"591e7d80-491c-4c58-802a-fc833b7ea25c\") "
Oct 30 19:57:20.516105 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.515894    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-credential-socket" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.533788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.522145    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-data" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.533788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.526525    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-certs" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.533788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.527141    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-data" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.533788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.528934    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-data" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.534942 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.534908    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-token" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.539828    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be62cbee-3784-4ec7-905c-f2751cf55f0e-istiod-ca-cert" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.542351    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-credential-socket" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.542434    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-certs" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.545339    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-credential-socket" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.545695    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4951a800-4431-4b8a-b6b7-2b400d641c2f-istiod-ca-cert" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.546056    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591e7d80-491c-4c58-802a-fc833b7ea25c-istiod-ca-cert" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.546318    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-socket" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.546559    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-certs" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.546673    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-token" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.547913    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-socket" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.548006    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-podinfo" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.548280    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-envoy" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.548404    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-kube-api-access-jl7xj" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "kube-api-access-jl7xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.549047    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-kube-api-access-z69vq" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "kube-api-access-z69vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.549860 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.549589    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-socket" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.551316 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.550704    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-podinfo" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:20.552946 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.552911    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-token" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.557955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.553219    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-podinfo" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:20.557955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.557801    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-envoy" pod "be62cbee-3784-4ec7-905c-f2751cf55f0e" (UID: "be62cbee-3784-4ec7-905c-f2751cf55f0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.557955 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.557866    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-envoy" pod "4951a800-4431-4b8a-b6b7-2b400d641c2f" (UID: "4951a800-4431-4b8a-b6b7-2b400d641c2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:20.558376 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.558357    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-kube-api-access-w9pgt" pod "591e7d80-491c-4c58-802a-fc833b7ea25c" (UID: "591e7d80-491c-4c58-802a-fc833b7ea25c"). InnerVolumeSpecName "kube-api-access-w9pgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.568212 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.567837    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608856    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608884    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608905    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608919    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9pgt\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-kube-api-access-w9pgt\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608933    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591e7d80-491c-4c58-802a-fc833b7ea25c-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608948    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608967    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be62cbee-3784-4ec7-905c-f2751cf55f0e-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608980    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.608993    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609005    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609018    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609032    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z69vq\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-kube-api-access-z69vq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609045    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609060    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609072    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591e7d80-491c-4c58-802a-fc833b7ea25c-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609085    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7xj\" (UniqueName: \"kubernetes.io/projected/4951a800-4431-4b8a-b6b7-2b400d641c2f-kube-api-access-jl7xj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609098    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951a800-4431-4b8a-b6b7-2b400d641c2f-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609123    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609136    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609149    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609162    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be62cbee-3784-4ec7-905c-f2751cf55f0e-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609175    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609189    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be62cbee-3784-4ec7-905c-f2751cf55f0e-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609202    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609215    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951a800-4431-4b8a-b6b7-2b400d641c2f-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609229    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.609292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.609242    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591e7d80-491c-4c58-802a-fc833b7ea25c-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:57:20.655642 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.655547    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh" event={"ID":"591e7d80-491c-4c58-802a-fc833b7ea25c","Type":"ContainerDied","Data":"41cd85a1aed10f8f58aed5b864757834b521598b0c5bff035c211f7f6dffe8ea"}
Oct 30 19:57:20.656493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.655911    4519 scope.go:122] "RemoveContainer" containerID="14c935898a5f231cda9d90fb94c96539039b10dc3380e6b26342512a85e00bdb"
Oct 30 19:57:20.656493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.656112    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"
Oct 30 19:57:20.677975 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.677848    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt" event={"ID":"be62cbee-3784-4ec7-905c-f2751cf55f0e","Type":"ContainerDied","Data":"b30e0acad8840143e5cc53ad518718672464b5019e2bf6f796dd89eae7831763"}
Oct 30 19:57:20.678144 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.677986    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"
Oct 30 19:57:20.681456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.681427    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t" event={"ID":"4951a800-4431-4b8a-b6b7-2b400d641c2f","Type":"ContainerDied","Data":"9baefddff8c4aee45a51ac2f9e1bc9623c561274afbdb5de11eeec2233209566"}
Oct 30 19:57:20.681621 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.681554    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"
Oct 30 19:57:20.687123 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.686904    4519 scope.go:122] "RemoveContainer" containerID="b194a079bc053415ea0f76ac9b92d86a965d9c85dc71dde808b3954d2d6627b4"
Oct 30 19:57:20.715686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.713724    4519 scope.go:122] "RemoveContainer" containerID="717cac305f3d8aa180c449d9f23564b1fcda0013912d27c8fb399a14f56d7550"
Oct 30 19:57:20.884091 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.884017    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"]
Oct 30 19:57:20.890667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.890578    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vhvhh"]
Oct 30 19:57:20.916021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.915947    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"]
Oct 30 19:57:20.921576 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.921510    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-tps4t"]
Oct 30 19:57:20.928160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.928122    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"]
Oct 30 19:57:20.939589 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.939453    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-rbklt"]
Oct 30 19:57:20.987322 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.987279    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:20.987456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.987405    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:20.987544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.987460    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:20.988425 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:20.988400    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"]
Oct 30 19:57:21.050028 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:57:21.049982    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6881d466_0536_4777_8072_0dc9d3733166.slice/crio-cee6b09d91ca226ad021b2666a264244d15a009f9e8fdd1f60dafeec5849f64b WatchSource:0}: Error finding container cee6b09d91ca226ad021b2666a264244d15a009f9e8fdd1f60dafeec5849f64b: Status 404 returned error can't find the container with id cee6b09d91ca226ad021b2666a264244d15a009f9e8fdd1f60dafeec5849f64b
Oct 30 19:57:21.062923 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.053335    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:21.063150 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.055037    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"]
Oct 30 19:57:21.063150 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.063085    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:21.063150 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.063144    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:21.679066 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.679027    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" path="/var/lib/kubelet/pods/4951a800-4431-4b8a-b6b7-2b400d641c2f/volumes"
Oct 30 19:57:21.680374 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.680345    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" path="/var/lib/kubelet/pods/591e7d80-491c-4c58-802a-fc833b7ea25c/volumes"
Oct 30 19:57:21.686410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.686381    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" path="/var/lib/kubelet/pods/be62cbee-3784-4ec7-905c-f2751cf55f0e/volumes"
Oct 30 19:57:21.727404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.727363    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" event={"ID":"6881d466-0536-4777-8072-0dc9d3733166","Type":"ContainerStarted","Data":"0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c"}
Oct 30 19:57:21.727731 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.727706    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" event={"ID":"6881d466-0536-4777-8072-0dc9d3733166","Type":"ContainerStarted","Data":"cee6b09d91ca226ad021b2666a264244d15a009f9e8fdd1f60dafeec5849f64b"}
Oct 30 19:57:21.789378 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.789339    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" event={"ID":"e89b6b48-a456-4ed3-b951-bb3d3dd498c1","Type":"ContainerStarted","Data":"991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204"}
Oct 30 19:57:21.789641 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.789620    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" event={"ID":"e89b6b48-a456-4ed3-b951-bb3d3dd498c1","Type":"ContainerStarted","Data":"dd3ab14897d6b7dea05572124514699a7527d18d38f529085f4a2e94b88aaaa6"}
Oct 30 19:57:21.848236 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.848167    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podStartSLOduration=2.8481440449999997 podStartE2EDuration="2.848144045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:21.839239433 +0000 UTC m=+1366.339943157" watchObservedRunningTime="2026-10-30 19:57:21.848144045 +0000 UTC m=+1366.348847768"
Oct 30 19:57:21.931156 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:21.931025    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podStartSLOduration=3.930990591 podStartE2EDuration="3.930990591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:21.897969701 +0000 UTC m=+1366.398673426" watchObservedRunningTime="2026-10-30 19:57:21.930990591 +0000 UTC m=+1366.431694317"
Oct 30 19:57:22.060492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.058751    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"]
Oct 30 19:57:22.085490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.083865    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b0b6a2-kln5l"]
Oct 30 19:57:22.154052 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.153953    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:22.220294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.219781    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:22.569099 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.569016    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:22.575419 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.575388    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:22.809264 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.809231    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:22.809812 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.809288    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:22.810423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.810158    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:57:22.813887 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:22.813866    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:57:23.618946 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:23.618907    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843d749e-6412-405b-81e1-35591f2e706d" path="/var/lib/kubelet/pods/843d749e-6412-405b-81e1-35591f2e706d/volumes"
Oct 30 19:57:23.821895 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:23.821804    4519 generic.go:356] "Generic (PLEG): container finished" podID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerID="2256ce950e5c68549704876d5f94a28c3b7c26a010c3899121a4d32118a3b60f" exitCode=0
Oct 30 19:57:23.822404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:23.821884    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" event={"ID":"40ba3d21-8105-42b2-bc93-abfb6953efca","Type":"ContainerDied","Data":"2256ce950e5c68549704876d5f94a28c3b7c26a010c3899121a4d32118a3b60f"}
Oct 30 19:57:23.822881 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:23.822858    4519 scope.go:122] "RemoveContainer" containerID="2256ce950e5c68549704876d5f94a28c3b7c26a010c3899121a4d32118a3b60f"
Oct 30 19:57:23.828782 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:23.828539    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:23.828782 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:23.828744    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:23.829096 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:23.828842    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:24.049488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:24.045203    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"]
Oct 30 19:57:24.053906 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:24.053863    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb59a549-d8fxz"]
Oct 30 19:57:24.868164 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:24.868129    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" event={"ID":"40ba3d21-8105-42b2-bc93-abfb6953efca","Type":"ContainerStarted","Data":"b6f382f3e9078a5ca769d79c02de50ec0d5dae90adb60444c76a0e3d7ef725db"}
Oct 30 19:57:25.050492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:25.049588    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"]
Oct 30 19:57:25.059494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:25.057955    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559c2a8f-tztfg"]
Oct 30 19:57:25.682162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:25.682125    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c202ad29-83e2-4a9b-8037-bcb5db160c6c" path="/var/lib/kubelet/pods/c202ad29-83e2-4a9b-8037-bcb5db160c6c/volumes"
Oct 30 19:57:25.683167 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:25.683141    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c2bb5e-7ebc-4605-9f3b-519c4d72f091" path="/var/lib/kubelet/pods/c3c2bb5e-7ebc-4605-9f3b-519c4d72f091/volumes"
Oct 30 19:57:25.683829 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:25.683807    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:57:25.718492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:25.706601    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:57:25.718492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:25.706675    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 19:57:26.712527 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:26.712397    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:57:26.714291 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:26.714247    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 19:57:27.433511 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.420510    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:57:27.598575 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.595016    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"]
Oct 30 19:57:27.698339 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.698244    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy"
Oct 30 19:57:27.698649 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.698626    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy"
Oct 30 19:57:27.698800 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.698787    4519 cpu_manager.go:414] "containerMap: removing container" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy"
Oct 30 19:57:27.698901 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.698882    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy"
Oct 30 19:57:27.699018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.699005    4519 cpu_manager.go:414] "containerMap: removing container" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy"
Oct 30 19:57:27.699120 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.699104    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy"
Oct 30 19:57:27.699604 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.699585    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4951a800-4431-4b8a-b6b7-2b400d641c2f" containerName="istio-proxy"
Oct 30 19:57:27.699749 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.699735    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="be62cbee-3784-4ec7-905c-f2751cf55f0e" containerName="istio-proxy"
Oct 30 19:57:27.699869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.699857    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="591e7d80-491c-4c58-802a-fc833b7ea25c" containerName="istio-proxy"
Oct 30 19:57:27.766516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.758017    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"]
Oct 30 19:57:27.806613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.781800    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.828798 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.828752    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0506c9-openshift-default-dockercfg-7sd8j\""
Oct 30 19:57:27.845112 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.845067    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-token\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.845392 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.845369    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istiod-ca-cert\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.845753 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.845724    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-certs\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.845913 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.845899    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.846142 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.846120    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-podinfo\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.852108 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.846255    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb74r\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-kube-api-access-hb74r\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.852455 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.852425    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-envoy\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.852721 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.852702    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-data\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.852905 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.852876    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-credential-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:27.928485 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.901220    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:57:27.928485 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.901307    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 19:57:27.953553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.944791    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.960864    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-certs\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.960936    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.961051    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-podinfo\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.961094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb74r\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-kube-api-access-hb74r\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.961136    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-envoy\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.961199    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-data\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.961233    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-credential-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.961442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-token\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.961501    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istiod-ca-cert\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.962660    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.962948    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-certs\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.998696    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-podinfo\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.076603 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:27.999344    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-credential-socket\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.123491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.103595    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istiod-ca-cert\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.157543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.157069    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:57:28.177870 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.158824    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:57:28.223717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.196268    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-token\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.223717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.198133    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:28.223717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.198855    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:57:28.294528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.244811    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"]
Oct 30 19:57:28.294528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.245160    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-data\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.294528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.271553    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc4068-n997c"]
Oct 30 19:57:28.294991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.294902    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:28.437896 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.437849    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 19:57:28.487256 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.487136    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-envoy\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.587191 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.587136    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb74r\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-kube-api-access-hb74r\") pod \"e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") " pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.654807 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.654329    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:57:28.654807 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.654393    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 19:57:28.908549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.897736    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:28.966493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.959258    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:57:28.966493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:28.960639    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:57:29.081045 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.080996    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:29.096819 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.096764    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 19:57:29.622380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.622289    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b98049-0de6-4d65-9050-a4949007a453" path="/var/lib/kubelet/pods/a6b98049-0de6-4d65-9050-a4949007a453/volumes"
Oct 30 19:57:29.692116 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.692075    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:57:29.692402 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.692371    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 19:57:29.811774 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.810704    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"]
Oct 30 19:57:29.812972 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.812939    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:29.813160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.813146    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:29.813250 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:29.813239    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:30.645456 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:30.645411    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" event={"ID":"5378b4fd-98d4-4fa8-88c3-31b19c705b45","Type":"ContainerStarted","Data":"b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d"}
Oct 30 19:57:30.645991 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:30.645610    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" event={"ID":"5378b4fd-98d4-4fa8-88c3-31b19c705b45","Type":"ContainerStarted","Data":"012afa46b5eac4b5f9d6ba5bef4c1d9028660551aa2bc2145f36fee064911806"}
Oct 30 19:57:30.666122 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:30.666078    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:57:30.666412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:30.666384    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 19:57:30.719953 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:30.719897    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podStartSLOduration=3.719882168 podStartE2EDuration="3.719882168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:30.715433268 +0000 UTC m=+1375.216136993" watchObservedRunningTime="2026-10-30 19:57:30.719882168 +0000 UTC m=+1375.220585893"
Oct 30 19:57:30.910964 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:30.910854    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:30.927892 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:30.927859    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:31.665920 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:31.665875    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:57:31.666639 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:31.666605    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 19:57:31.687486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:31.687428    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:31.688403 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:31.688382    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:57:32.252363 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:57:32.252320    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:57:32.655328 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:32.655229    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:57:32.658867 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:32.656892    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 19:57:39.057683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:39.057641    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"]
Oct 30 19:57:39.072865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:39.072821    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e752e050-4zndq"]
Oct 30 19:57:39.617095 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:39.617053    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d95dd9-8d43-404e-9625-fa688dabef01" path="/var/lib/kubelet/pods/f5d95dd9-8d43-404e-9625-fa688dabef01/volumes"
Oct 30 19:57:40.025522 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:40.025487    4519 scope.go:122] "RemoveContainer" containerID="88363467deb7b23ce6a67db8c3ebf1604a2063e3a3f2a2870846764be752068c"
Oct 30 19:57:40.037906 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:40.037879    4519 scope.go:122] "RemoveContainer" containerID="20119578225c321e2db83808ea324e9ff0636a12a467501d1f030b8c9d714591"
Oct 30 19:57:40.050015 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:40.049996    4519 scope.go:122] "RemoveContainer" containerID="1a900b66f938659340d4a0a00b282e0af198458c400f0d309c1f0d04abf7caea"
Oct 30 19:57:40.061812 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:40.061781    4519 scope.go:122] "RemoveContainer" containerID="3fb2cf00bcaa7f6ef37eed71e9b03278baf87eda15fdc1e75862ac28081d309a"
Oct 30 19:57:40.073659 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:40.073633    4519 scope.go:122] "RemoveContainer" containerID="98b75d6163f05b2720f52496cdec17be54914e454bc19794ff67c6035b9f709b"
Oct 30 19:57:40.086130 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:40.086104    4519 scope.go:122] "RemoveContainer" containerID="95a5c2859c9aba2a8cecfcbd28e416060dd8259048d4251800f73fca8569912d"
Oct 30 19:57:44.033284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:44.033247    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"]
Oct 30 19:57:44.045588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:44.045547    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff20d7e7-f4nfv"]
Oct 30 19:57:45.035559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:45.035514    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"]
Oct 30 19:57:45.048744 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:45.048701    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"]
Oct 30 19:57:45.056978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:45.056937    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cce0870-rw784"]
Oct 30 19:57:45.065218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:45.065179    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0eef37-77knl"]
Oct 30 19:57:45.616882 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:45.616849    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161c49f7-8406-4566-9703-7f76c8ed443b" path="/var/lib/kubelet/pods/161c49f7-8406-4566-9703-7f76c8ed443b/volumes"
Oct 30 19:57:45.617429 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:45.617411    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f073363-f521-4176-aae3-7b146033048b" path="/var/lib/kubelet/pods/9f073363-f521-4176-aae3-7b146033048b/volumes"
Oct 30 19:57:45.617894 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:45.617879    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe8f850-4a29-45ec-9068-4072c8859aa7" path="/var/lib/kubelet/pods/bbe8f850-4a29-45ec-9068-4072c8859aa7/volumes"
Oct 30 19:57:49.372497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.372381    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"]
Oct 30 19:57:49.379305 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.379267    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.407950 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.407914    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"]
Oct 30 19:57:49.486327 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.486287    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.486543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.486334    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a79d6-7154-4e61-a0de-d5886a1e6726-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.486543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.486440    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.486683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.486669    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbcp\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-kube-api-access-sxbcp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.486725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.486714    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.486847 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.486834    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.486896 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.486877    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.588333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.588286    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbcp\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-kube-api-access-sxbcp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.588574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.588357    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.588574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.588476    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.588574 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.588522    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.588786 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.588574    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.588786 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.588603    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a79d6-7154-4e61-a0de-d5886a1e6726-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.588786 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.588642    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.590038 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.590003    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.591153 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.591120    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.591671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.591644    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.591737 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.591682    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a79d6-7154-4e61-a0de-d5886a1e6726-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.592024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.592002    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.608134 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.608095    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.608313 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.608286    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbcp\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-kube-api-access-sxbcp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cggfl\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:49.695411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:49.695364    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:50.029408 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:57:50.029374    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5a79d6_7154_4e61_a0de_d5886a1e6726.slice/crio-343120a0e295be9c7c8ae5b1b83925d62dd4b4f37c41f3b3eee53d5df0feb6ee WatchSource:0}: Error finding container 343120a0e295be9c7c8ae5b1b83925d62dd4b4f37c41f3b3eee53d5df0feb6ee: Status 404 returned error can't find the container with id 343120a0e295be9c7c8ae5b1b83925d62dd4b4f37c41f3b3eee53d5df0feb6ee
Oct 30 19:57:50.032528 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:50.032458    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:50.032666 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:50.032570    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:50.039825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:50.035529    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"]
Oct 30 19:57:50.899162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:50.899068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl" event={"ID":"8f5a79d6-7154-4e61-a0de-d5886a1e6726","Type":"ContainerStarted","Data":"c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5"}
Oct 30 19:57:50.899162 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:50.899143    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl" event={"ID":"8f5a79d6-7154-4e61-a0de-d5886a1e6726","Type":"ContainerStarted","Data":"343120a0e295be9c7c8ae5b1b83925d62dd4b4f37c41f3b3eee53d5df0feb6ee"}
Oct 30 19:57:50.899742 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:50.899178    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:50.969777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:50.969709    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl" podStartSLOduration=1.9696915499999998 podStartE2EDuration="1.96969155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:50.967373487 +0000 UTC m=+1395.468077202" watchObservedRunningTime="2026-10-30 19:57:50.96969155 +0000 UTC m=+1395.470395274"
Oct 30 19:57:51.915412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:51.915372    4519 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cggfl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:57:51.915853 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:51.915435    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl" podUID="8f5a79d6-7154-4e61-a0de-d5886a1e6726" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:57:52.045144 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:52.045106    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"]
Oct 30 19:57:52.060109 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:52.060066    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ab4f53-mnvpq"]
Oct 30 19:57:53.745175 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:53.738278    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757dd504-0e6f-4c43-849e-d73c32288a30" path="/var/lib/kubelet/pods/757dd504-0e6f-4c43-849e-d73c32288a30/volumes"
Oct 30 19:57:54.915486 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:54.915245    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 19:57:55.040733 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:55.040690    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"]
Oct 30 19:57:55.047107 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:55.047070    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a559c5-pqctj"]
Oct 30 19:57:55.615029 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:55.614975    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1600c0-ab78-4328-9ddf-35d06f693e77" path="/var/lib/kubelet/pods/4d1600c0-ab78-4328-9ddf-35d06f693e77/volumes"
Oct 30 19:57:57.070488 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:57.069952    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"]
Oct 30 19:57:57.078116 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:57.078073    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447b8c9-cjjvw"]
Oct 30 19:57:57.681517 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:57.681430    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db124ff-ad32-4eff-93c9-2704963b43b4" path="/var/lib/kubelet/pods/5db124ff-ad32-4eff-93c9-2704963b43b4/volumes"
Oct 30 19:57:59.235683 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:59.226224    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"]
Oct 30 19:57:59.401493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:59.377563    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3231786e-sbk7n"]
Oct 30 19:57:59.633716 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:57:59.633640    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112f5ed2-38fc-4ac8-9da1-ddaa221551b9" path="/var/lib/kubelet/pods/112f5ed2-38fc-4ac8-9da1-ddaa221551b9/volumes"
Oct 30 19:58:01.113005 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:01.112959    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:01.121735 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:01.121628    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:01.270233 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:01.270099    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:02.411008 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:58:02.410962    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:58:09.609512 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:09.609454    4519 patch_prober.go:29] interesting pod/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:58:09.610160 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:09.610117    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:11.037550 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:11.037502    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"]
Oct 30 19:58:11.045805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:11.045763    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83440e29-848lz"]
Oct 30 19:58:11.636331 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:11.636283    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afba94c2-b3ef-4800-91f0-8aa02e2e4785" path="/var/lib/kubelet/pods/afba94c2-b3ef-4800-91f0-8aa02e2e4785/volumes"
Oct 30 19:58:12.020503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:12.011964    4519 generic.go:356] "Generic (PLEG): container finished" podID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerID="f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0" exitCode=0
Oct 30 19:58:12.020503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:12.012261    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" event={"ID":"d00ba69a-0c7c-45da-9e5d-16b26139248b","Type":"ContainerDied","Data":"f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0"}
Oct 30 19:58:12.020503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:12.012948    4519 scope.go:122] "RemoveContainer" containerID="f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0"
Oct 30 19:58:12.030727 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:12.030674    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:12.031650 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:12.031610    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:12.031865 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:12.031844    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:13.025298 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.025257    4519 generic.go:356] "Generic (PLEG): container finished" podID="2fc2d597-f684-4579-ba47-7735c112706a" containerID="76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977" exitCode=0
Oct 30 19:58:13.025898 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.025329    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" event={"ID":"2fc2d597-f684-4579-ba47-7735c112706a","Type":"ContainerDied","Data":"76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977"}
Oct 30 19:58:13.025966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.025930    4519 scope.go:122] "RemoveContainer" containerID="76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977"
Oct 30 19:58:13.028025 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.028000    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" event={"ID":"d00ba69a-0c7c-45da-9e5d-16b26139248b","Type":"ContainerStarted","Data":"c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14"}
Oct 30 19:58:13.028154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.028039    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:13.028154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.028111    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:13.028154 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.028143    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:13.722353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.722319    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:58:13.743199 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.731631    4519 patch_prober.go:29] interesting pod/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 19:58:13.743572 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:13.743536    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 19:58:14.090553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:14.077704    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" event={"ID":"2fc2d597-f684-4579-ba47-7735c112706a","Type":"ContainerStarted","Data":"0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb"}
Oct 30 19:58:14.722592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:14.722553    4519 patch_prober.go:29] interesting pod/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 19:58:14.722789 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:14.722614    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 19:58:14.930284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:14.930247    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:58:14.930763 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:14.930735    4519 patch_prober.go:29] interesting pod/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:58:14.931000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:14.930973    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 19:58:15.722938 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:15.722894    4519 patch_prober.go:29] interesting pod/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 19:58:15.723571 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:15.723534    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 19:58:15.927978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:15.927935    4519 patch_prober.go:29] interesting pod/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:58:15.928189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:15.927997    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 19:58:16.721516 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:16.721448    4519 patch_prober.go:29] interesting pod/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 19:58:16.721708 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:16.721546    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 19:58:16.929032 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:16.928992    4519 patch_prober.go:29] interesting pod/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:58:16.929438 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:16.929049    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 19:58:17.721083 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:17.721045    4519 patch_prober.go:29] interesting pod/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 19:58:17.721280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:17.721104    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 19:58:17.928103 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:17.928061    4519 patch_prober.go:29] interesting pod/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:58:17.928270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:17.928135    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 19:58:18.031172 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:18.031070    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"]
Oct 30 19:58:18.043130 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:18.043092    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e74899-g9cqx"]
Oct 30 19:58:18.721725 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:18.721685    4519 patch_prober.go:29] interesting pod/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 19:58:18.721914 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:18.721782    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 19:58:18.928110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:18.928078    4519 patch_prober.go:29] interesting pod/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:58:18.928328 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:18.928129    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 19:58:19.614014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:19.613970    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041aad9a-34bc-4633-9fe6-a57d8b662011" path="/var/lib/kubelet/pods/041aad9a-34bc-4633-9fe6-a57d8b662011/volumes"
Oct 30 19:58:19.726646 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:19.726615    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:58:19.727205 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:19.727183    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:58:19.727774 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:19.727755    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 19:58:19.928136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:19.928102    4519 patch_prober.go:29] interesting pod/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:58:19.928334 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:19.928171    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 19:58:20.478816 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:20.478777    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:58:20.943526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:20.942631    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:58:21.033374 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:21.033337    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"]
Oct 30 19:58:21.043569 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:21.043534    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3728ca80-2rf82"]
Oct 30 19:58:21.240483 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:21.240391    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 19:58:21.615200 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:21.615103    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65eeb886-ac6c-42c5-8cb4-00319cb62ae2" path="/var/lib/kubelet/pods/65eeb886-ac6c-42c5-8cb4-00319cb62ae2/volumes"
Oct 30 19:58:23.032667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:23.032630    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"]
Oct 30 19:58:23.044709 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:23.044675    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246f440f-qc9xb"]
Oct 30 19:58:23.614815 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:23.614784    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8" path="/var/lib/kubelet/pods/d5c2d0cc-9565-45de-9cb0-f214c8aa6fc8/volumes"
Oct 30 19:58:28.531491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:28.528590    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"]
Oct 30 19:58:28.531491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:28.528911    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy" containerID="cri-o://81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef" gracePeriod=30
Oct 30 19:58:28.794559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:28.793058    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"]
Oct 30 19:58:28.794559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:28.793350    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" containerID="cri-o://17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359" gracePeriod=30
Oct 30 19:58:31.413609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:31.413550    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:58:31.414015 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:31.413633    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:58:32.315263 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:58:32.315176    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:58:34.318140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.317694    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:58:34.343069 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343032    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-kube-api-access-zrctr\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-kube-api-access-zrctr\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343260 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343085    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-podinfo\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343260 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343137    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-credential-socket\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343400 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343340    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-token\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-token\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343505 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343487    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-socket\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343590 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343553    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istiod-ca-cert\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343699    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-envoy\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343731    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-credential-socket" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.343787 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343744    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-certs\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343925 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343884    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-data\") pod \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\" (UID: \"6b0f7b52-9b85-48a0-bf10-0926ad6db89e\") "
Oct 30 19:58:34.343990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.343949    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-certs" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.344280 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.344257    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-socket" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.344346 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.344190    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istiod-ca-cert" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:34.344510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.344480    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-data" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.344945 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.344921    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.345314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.345296    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.345451 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.345438    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.345580 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.345566    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.345679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.345663    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.387491 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.374801    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-token" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:34.412577 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.401719    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-kube-api-access-zrctr" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "kube-api-access-zrctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:34.418769 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.418634    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-envoy" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.418769 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.418738    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-podinfo" pod "6b0f7b52-9b85-48a0-bf10-0926ad6db89e" (UID: "6b0f7b52-9b85-48a0-bf10-0926ad6db89e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:34.471547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.469161    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.471547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.469199    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.471547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.469215    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrctr\" (UniqueName: \"kubernetes.io/projected/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-kube-api-access-zrctr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.471547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.469230    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0f7b52-9b85-48a0-bf10-0926ad6db89e-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.681692 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.681657    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.720194    4519 generic.go:356] "Generic (PLEG): container finished" podID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerID="17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359" exitCode=0
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.720268    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" event={"ID":"242639b8-6ae8-4a9f-9744-ddcccb88b59b","Type":"ContainerDied","Data":"17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359"}
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.720324    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh" event={"ID":"242639b8-6ae8-4a9f-9744-ddcccb88b59b","Type":"ContainerDied","Data":"b77d1f01bc6f2347b87e44d5850c1cd2b2918c8717ea5d9fc85a3f94b923c91a"}
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.720349    4519 scope.go:122] "RemoveContainer" containerID="17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359"
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.720455    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.722034    4519 generic.go:356] "Generic (PLEG): container finished" podID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerID="81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef" exitCode=0
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.722121    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" event={"ID":"6b0f7b52-9b85-48a0-bf10-0926ad6db89e","Type":"ContainerDied","Data":"81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef"}
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.722152    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf" event={"ID":"6b0f7b52-9b85-48a0-bf10-0926ad6db89e","Type":"ContainerDied","Data":"1765a6d5921dba4fdf6eea9238b9cc13b8eafe31e3db4f098580dcd1cada39d1"}
Oct 30 19:58:34.726562 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.722202    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"
Oct 30 19:58:34.739668 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.739622    4519 scope.go:122] "RemoveContainer" containerID="17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359"
Oct 30 19:58:34.741549 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:58:34.741520    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359\": container with ID starting with 17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359 not found: ID does not exist" containerID="17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359"
Oct 30 19:58:34.741719 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.741687    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359"} err="failed to get container status \"17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359\": rpc error: code = NotFound desc = could not find container \"17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359\": container with ID starting with 17647293426dfbb24d8ce0bbfd4ac70d71b9fe37acc67e8ce36cb2a1f22ea359 not found: ID does not exist"
Oct 30 19:58:34.741719 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.741718    4519 scope.go:122] "RemoveContainer" containerID="81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef"
Oct 30 19:58:34.816435 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.816410    4519 scope.go:122] "RemoveContainer" containerID="81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef"
Oct 30 19:58:34.817021 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:58:34.816996    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef\": container with ID starting with 81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef not found: ID does not exist" containerID="81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef"
Oct 30 19:58:34.817181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.817152    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef"} err="failed to get container status \"81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef\": rpc error: code = NotFound desc = could not find container \"81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef\": container with ID starting with 81e58d2d50e450d33c4c451553e75756e5287b61ad7b67bdecc9406f5fc978ef not found: ID does not exist"
Oct 30 19:58:34.834444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.834421    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-credential-socket\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.834847 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.834823    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istiod-ca-cert\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.835024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835010    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-envoy\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.835404 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835389    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-kube-api-access-m8wdb\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-kube-api-access-m8wdb\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.835579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835561    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-certs\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.835663 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.834919    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-credential-socket" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.835718 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835241    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istiod-ca-cert" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:34.835718 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835671    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-token\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-token\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.835883 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835744    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-socket\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.835883 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835779    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-data\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.835984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835879    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-certs" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.835984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.835897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-podinfo\") pod \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\" (UID: \"242639b8-6ae8-4a9f-9744-ddcccb88b59b\") "
Oct 30 19:58:34.836082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.836062    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-socket" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.836287 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.836267    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-data" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.837014 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.836995    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.837101 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.837017    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.837101 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.837031    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.837101 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.837043    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.837101 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.837056    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.837575 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.837552    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-envoy" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:34.838118 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.838094    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-kube-api-access-m8wdb" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "kube-api-access-m8wdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:34.838314 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.838288    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-token" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:34.838775 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.838756    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-podinfo" pod "242639b8-6ae8-4a9f-9744-ddcccb88b59b" (UID: "242639b8-6ae8-4a9f-9744-ddcccb88b59b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:34.931188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:34.930921    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"]
Oct 30 19:58:35.017556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.004031    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:35.017556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.004075    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wdb\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-kube-api-access-m8wdb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:35.017556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.004095    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:35.017556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.004110    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242639b8-6ae8-4a9f-9744-ddcccb88b59b-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:35.025530 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.025485    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p5jhf"]
Oct 30 19:58:35.435337 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.435296    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"]
Oct 30 19:58:35.832544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.829713    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" path="/var/lib/kubelet/pods/6b0f7b52-9b85-48a0-bf10-0926ad6db89e/volumes"
Oct 30 19:58:35.832544 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:35.830200    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-p58lh"]
Oct 30 19:58:37.615748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:37.615650    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" path="/var/lib/kubelet/pods/242639b8-6ae8-4a9f-9744-ddcccb88b59b/volumes"
Oct 30 19:58:38.851044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:38.850643    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:58:38.851044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:38.850712    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:40.213053 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.212902    4519 scope.go:122] "RemoveContainer" containerID="67f6c07596e6ffcfbeab13ef716be411af19c3ab300af1c987ddbba243996823"
Oct 30 19:58:40.231892 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.228988    4519 scope.go:122] "RemoveContainer" containerID="d25592d11101637f79b979a6e5eb7bab1310eea72332e25dde98569ef01bb74c"
Oct 30 19:58:40.243704 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.243676    4519 scope.go:122] "RemoveContainer" containerID="be21b0f5678cf669d6079f5a4046c81d5cc5622b851fd9f685246bb6fb99c9b7"
Oct 30 19:58:40.294412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.294385    4519 scope.go:122] "RemoveContainer" containerID="272c9b9c43a41b9512fde245dcd1fff30a36ffacd410ad593d2c9086863c55b5"
Oct 30 19:58:40.328021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.327990    4519 scope.go:122] "RemoveContainer" containerID="05da1f7d3bed812b59d65e0422990f6c6978156c083391368cf452992891de45"
Oct 30 19:58:40.357922 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.357888    4519 scope.go:122] "RemoveContainer" containerID="cc3b7ff38028bef950dd36692364ad5c2c54837cbb9ea02460015d717636a27c"
Oct 30 19:58:40.399997 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.399970    4519 scope.go:122] "RemoveContainer" containerID="4f82fe36c230e62716d968d0c6cede75105fe1a469e9185e1ede07cb6aef6f41"
Oct 30 19:58:40.432082 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.432052    4519 scope.go:122] "RemoveContainer" containerID="02053e683df865447c726ca4c3f4da0e85fc2d0d5e8791a52a1877e57287b0c3"
Oct 30 19:58:40.470427 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.469671    4519 scope.go:122] "RemoveContainer" containerID="4cb03f210da33dae1e2e38afd144bc09ea40ba7b68d5cba40b652d396e270ab6"
Oct 30 19:58:40.498874 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.498842    4519 scope.go:122] "RemoveContainer" containerID="49d1c06c306cadca028f4f68e64501c071bf4daaec9263c552950efbf6bda031"
Oct 30 19:58:40.511292 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.511266    4519 scope.go:122] "RemoveContainer" containerID="8dc70d0691f73f87dfbc425cb04796d576145beae884aed011d232e1273b3d2c"
Oct 30 19:58:40.523521 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.523484    4519 scope.go:122] "RemoveContainer" containerID="af99783110721d657644cfb557cb556449521be67a33d84c6fdd395d2bc15464"
Oct 30 19:58:40.537982 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:40.537944    4519 scope.go:122] "RemoveContainer" containerID="86c45e005279d74a2591adbfd763fe2a4cd00519afedf655bd80eba9d727831d"
Oct 30 19:58:42.019833 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.019785    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"]
Oct 30 19:58:42.021607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.020994    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy"
Oct 30 19:58:42.021607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.021022    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy"
Oct 30 19:58:42.021607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.021088    4519 cpu_manager.go:414] "containerMap: removing container" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy"
Oct 30 19:58:42.021607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.021097    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy"
Oct 30 19:58:42.021607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.021262    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="242639b8-6ae8-4a9f-9744-ddcccb88b59b" containerName="istio-proxy"
Oct 30 19:58:42.021607 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.021279    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b0f7b52-9b85-48a0-bf10-0926ad6db89e" containerName="istio-proxy"
Oct 30 19:58:42.065504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.062837    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.102270 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.094298    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-87ff9f-openshift-default-dockercfg-qxv5h\""
Oct 30 19:58:42.156974 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.134817    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"]
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.184582    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-credential-socket\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.184675    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-data\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.184916    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-socket\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.184982    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istiod-ca-cert\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.185057    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-podinfo\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.185126    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-envoy\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.185152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxljp\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-kube-api-access-kxljp\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.185184    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-token\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.198957 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.185226    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-certs\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.432492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.421224    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-socket\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.444708    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istiod-ca-cert\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.444857    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-podinfo\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.444944    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-envoy\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.444973    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxljp\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-kube-api-access-kxljp\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.445007    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-token\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.445056    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-certs\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.445096    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-credential-socket\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.448497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.445173    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-data\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.459503 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.450934    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-socket\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.579492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.465200    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-credential-socket\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.579492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.465519    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-certs\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.579492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.465786    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-data\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.579492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.500921    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istiod-ca-cert\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.579492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.501367    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxljp\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-kube-api-access-kxljp\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.579492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.550088    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-podinfo\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.664383 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.650922    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-envoy\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.664383 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.653485    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-token\") pod \"e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") " pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:42.971554 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:42.968026    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:43.087387 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.066739    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:58:43.087387 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.077771    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:43.141492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.124509    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:43.203809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.135317    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:43.203809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.135356    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:43.203809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.155590    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:43.203809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.155679    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:58:43.203809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.155707    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:58:44.001780 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:43.993305    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"]
Oct 30 19:58:44.004948 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:58:44.004892    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79bd922_ebb1_48e9_a300_9e0efb1b69d3.slice/crio-28616ed13723808366bcb36565691ebe99fdc585ca1b4a308e6ec7accee103ed WatchSource:0}: Error finding container 28616ed13723808366bcb36565691ebe99fdc585ca1b4a308e6ec7accee103ed: Status 404 returned error can't find the container with id 28616ed13723808366bcb36565691ebe99fdc585ca1b4a308e6ec7accee103ed
Oct 30 19:58:44.013065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:44.013013    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:44.013234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:44.013121    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:44.013234 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:44.013166    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:44.735852 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:44.734735    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" event={"ID":"d79bd922-ebb1-48e9-a300-9e0efb1b69d3","Type":"ContainerStarted","Data":"a5212b395dd59dc1607dac435235de9815e5129ecedcaa7e9c202e660c3ce5d3"}
Oct 30 19:58:44.735852 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:44.734815    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" event={"ID":"d79bd922-ebb1-48e9-a300-9e0efb1b69d3","Type":"ContainerStarted","Data":"28616ed13723808366bcb36565691ebe99fdc585ca1b4a308e6ec7accee103ed"}
Oct 30 19:58:44.827697 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:44.827621    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podStartSLOduration=3.827598832 podStartE2EDuration="3.827598832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:44.809723107 +0000 UTC m=+1449.310426833" watchObservedRunningTime="2026-10-30 19:58:44.827598832 +0000 UTC m=+1449.328302558"
Oct 30 19:58:44.984519 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:44.984042    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:45.634452 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.634393    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"]
Oct 30 19:58:45.639717 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.639679    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.664729 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.664664    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:45.664926 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.664821    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:58:45.702739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.702696    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"]
Oct 30 19:58:45.797497 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.787649    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.798194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.798157    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znn95\" (UniqueName: \"kubernetes.io/projected/e577efa4-43e6-420d-852a-f003d06b4737-kube-api-access-znn95\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.798588 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.798563    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.901534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.901419    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.903606 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.903390    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.903876 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.903849    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znn95\" (UniqueName: \"kubernetes.io/projected/e577efa4-43e6-420d-852a-f003d06b4737-kube-api-access-znn95\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.904362 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.904305    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.905020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.904995    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.929631 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.929592    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znn95\" (UniqueName: \"kubernetes.io/projected/e577efa4-43e6-420d-852a-f003d06b4737-kube-api-access-znn95\") pod \"maas-api-revoke-sub-f3304bad-5vw45\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:45.985480 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.985426    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:46.000553 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:45.998912    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:46.423514 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:46.423331    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"]
Oct 30 19:58:46.424181 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:58:46.424148    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode577efa4_43e6_420d_852a_f003d06b4737.slice/crio-8e0fb126a4624765def456e11de96704926bba8e15efcdc993d879c5dae14a0a WatchSource:0}: Error finding container 8e0fb126a4624765def456e11de96704926bba8e15efcdc993d879c5dae14a0a: Status 404 returned error can't find the container with id 8e0fb126a4624765def456e11de96704926bba8e15efcdc993d879c5dae14a0a
Oct 30 19:58:46.807722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:46.807592    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45" event={"ID":"e577efa4-43e6-420d-852a-f003d06b4737","Type":"ContainerStarted","Data":"8e0fb126a4624765def456e11de96704926bba8e15efcdc993d879c5dae14a0a"}
Oct 30 19:58:46.815622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:46.807838    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:46.815622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:46.808974    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 19:58:47.045623 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:47.045586    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": read tcp 10.133.0.2:39794->10.133.0.161:15021: read: connection reset by peer" start-of-body=
Oct 30 19:58:47.045878 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:47.045836    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": read tcp 10.133.0.2:39794->10.133.0.161:15021: read: connection reset by peer"
Oct 30 19:58:48.044518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.044352    4519 generic.go:356] "Generic (PLEG): container finished" podID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerID="991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204" exitCode=0
Oct 30 19:58:48.045125 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.044625    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" event={"ID":"e89b6b48-a456-4ed3-b951-bb3d3dd498c1","Type":"ContainerDied","Data":"991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204"}
Oct 30 19:58:48.056788 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.056745    4519 scope.go:122] "RemoveContainer" containerID="991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204"
Oct 30 19:58:48.097100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.090804    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:48.097100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.090920    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:48.097100 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.090970    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:48.210585 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.202159    4519 generic.go:356] "Generic (PLEG): container finished" podID="e577efa4-43e6-420d-852a-f003d06b4737" containerID="bb6a02a206b285ceb21f27d8dc75b0751f86c7f841caea2b4c228778ccb725c0" exitCode=0
Oct 30 19:58:48.210585 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.202344    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45" event={"ID":"e577efa4-43e6-420d-852a-f003d06b4737","Type":"ContainerDied","Data":"bb6a02a206b285ceb21f27d8dc75b0751f86c7f841caea2b4c228778ccb725c0"}
Oct 30 19:58:48.226368 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.226328    4519 generic.go:356] "Generic (PLEG): container finished" podID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerID="b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d" exitCode=0
Oct 30 19:58:48.227114 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.226640    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" event={"ID":"5378b4fd-98d4-4fa8-88c3-31b19c705b45","Type":"ContainerDied","Data":"b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d"}
Oct 30 19:58:48.227888 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.227859    4519 scope.go:122] "RemoveContainer" containerID="b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d"
Oct 30 19:58:48.231927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.230307    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:48.231927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.230436    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:48.231927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:48.230504    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:49.265592 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:49.265546    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" event={"ID":"5378b4fd-98d4-4fa8-88c3-31b19c705b45","Type":"ContainerStarted","Data":"dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0"}
Oct 30 19:58:49.302412 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:49.302360    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" event={"ID":"e89b6b48-a456-4ed3-b951-bb3d3dd498c1","Type":"ContainerStarted","Data":"428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c"}
Oct 30 19:58:50.030548 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:49.931658    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:58:50.405558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.251583    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:58:50.611494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.545651    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:58:50.611494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.545722    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 19:58:50.611494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.546618    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:58:50.611494 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.546663    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 19:58:50.722686 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.722651    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:50.804492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.804253    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-openshift-ca\") pod \"e577efa4-43e6-420d-852a-f003d06b4737\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") "
Oct 30 19:58:50.804711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.804601    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e577efa4-43e6-420d-852a-f003d06b4737-kube-api-access-znn95\" (UniqueName: \"kubernetes.io/projected/e577efa4-43e6-420d-852a-f003d06b4737-kube-api-access-znn95\") pod \"e577efa4-43e6-420d-852a-f003d06b4737\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") "
Oct 30 19:58:50.827490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.804785    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-xks-ca\") pod \"e577efa4-43e6-420d-852a-f003d06b4737\" (UID: \"e577efa4-43e6-420d-852a-f003d06b4737\") "
Oct 30 19:58:50.827490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.808952    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-openshift-ca" pod "e577efa4-43e6-420d-852a-f003d06b4737" (UID: "e577efa4-43e6-420d-852a-f003d06b4737"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:50.830492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.829627    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"
Oct 30 19:58:50.830675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.830608    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45" event={"ID":"e577efa4-43e6-420d-852a-f003d06b4737","Type":"ContainerDied","Data":"8e0fb126a4624765def456e11de96704926bba8e15efcdc993d879c5dae14a0a"}
Oct 30 19:58:50.830675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.830663    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0fb126a4624765def456e11de96704926bba8e15efcdc993d879c5dae14a0a"
Oct 30 19:58:50.844662 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.844623    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-xks-ca" pod "e577efa4-43e6-420d-852a-f003d06b4737" (UID: "e577efa4-43e6-420d-852a-f003d06b4737"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:50.846357 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.846327    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e577efa4-43e6-420d-852a-f003d06b4737-kube-api-access-znn95" pod "e577efa4-43e6-420d-852a-f003d06b4737" (UID: "e577efa4-43e6-420d-852a-f003d06b4737"). InnerVolumeSpecName "kube-api-access-znn95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:50.915647 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.915602    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znn95\" (UniqueName: \"kubernetes.io/projected/e577efa4-43e6-420d-852a-f003d06b4737-kube-api-access-znn95\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:50.916245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.916221    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:50.916395 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.916381    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e577efa4-43e6-420d-852a-f003d06b4737-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:58:50.924537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.923642    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:58:50.924537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:50.923713    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 19:58:51.159900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:51.137929    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:58:51.159900 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:51.138010    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 19:58:51.913714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:51.904892    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:58:51.913714 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:51.904960    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 19:58:52.123990 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:52.123946    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:58:52.124194 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:52.124015    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 19:58:52.810648 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:52.810604    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:58:52.903961 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:52.903917    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:58:52.904277 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:52.904244    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 19:58:53.123431 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:53.123312    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:58:53.124012 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:53.123980    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 19:58:53.864492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:53.859670    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:58:53.864492 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:53.859761    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:53.917830 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:53.917792    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:58:53.918020 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:53.917856    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 19:58:54.129549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:54.120833    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:58:54.129549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:54.120909    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 19:58:54.913966 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:54.913923    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:58:54.914305 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:54.914267    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 19:58:55.128507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.123813    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:58:55.128507 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.123880    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 19:58:55.770490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.767800    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"]
Oct 30 19:58:55.789986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.784324    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e577efa4-43e6-420d-852a-f003d06b4737" containerName="revoke-keys"
Oct 30 19:58:55.789986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.784361    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e577efa4-43e6-420d-852a-f003d06b4737" containerName="revoke-keys"
Oct 30 19:58:55.789986 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.784632    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e577efa4-43e6-420d-852a-f003d06b4737" containerName="revoke-keys"
Oct 30 19:58:55.847692 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.847616    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:55.859995 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.859929    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"]
Oct 30 19:58:55.908936 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.898986    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-eb16ce-openshift-default-dockercfg-x2mq2\""
Oct 30 19:58:55.958584 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.911134    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:58:55.958584 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:55.938116    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 19:58:56.181560 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.181523    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-certs\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.182021 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.181991    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-credential-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.182278 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.182259    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istiod-ca-cert\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.249645    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-data\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.249789    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-token\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.249931    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.250005    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5xxl\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-kube-api-access-z5xxl\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.250092    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-podinfo\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.250154    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-envoy\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.252879    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:58:56.255490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.252959    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 19:58:56.418489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.414869    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.454722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.454614    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5xxl\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-kube-api-access-z5xxl\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.455099 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.455072    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-podinfo\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.455333 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.455312    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-envoy\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.455510 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.455491    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-certs\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.455841 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.455820    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-credential-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.456110 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.456080    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istiod-ca-cert\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.456252 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.456234    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-data\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.456444 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.456431    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-token\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.466493 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.424990    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.486178 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.460932    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-certs\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.486447 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.461059    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-credential-socket\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.492113 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.461849    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istiod-ca-cert\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.492410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.462123    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-data\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.562889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.547433    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5xxl\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-kube-api-access-z5xxl\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.583417 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.583033    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-envoy\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.590077 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.590039    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-token\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.616542 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.615320    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-podinfo\") pod \"e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") " pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:56.858049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:56.857948    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:57.149558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.147200    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:58:57.180060 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.178502    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 19:58:57.247532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.244632    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:58:57.247532 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.246024    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:58:57.249050 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.249014    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 19:58:57.296217 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:58:57.296178    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cf99a9_cef2_4e0b_8455_47bbd2c33edf.slice/crio-4bece96dfcf34d4801012450f7fddd3471796848cd8d88c78a50ae1b50ba117d WatchSource:0}: Error finding container 4bece96dfcf34d4801012450f7fddd3471796848cd8d88c78a50ae1b50ba117d: Status 404 returned error can't find the container with id 4bece96dfcf34d4801012450f7fddd3471796848cd8d88c78a50ae1b50ba117d
Oct 30 19:58:57.301839 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.301793    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:57.302000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.301896    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:57.302000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.301946    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:57.324340 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.324299    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"]
Oct 30 19:58:57.552962 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.552925    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" event={"ID":"11cf99a9-cef2-4e0b-8455-47bbd2c33edf","Type":"ContainerStarted","Data":"0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405"}
Oct 30 19:58:57.553218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.553199    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" event={"ID":"11cf99a9-cef2-4e0b-8455-47bbd2c33edf","Type":"ContainerStarted","Data":"4bece96dfcf34d4801012450f7fddd3471796848cd8d88c78a50ae1b50ba117d"}
Oct 30 19:58:57.608330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.606020    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podStartSLOduration=2.606003168 podStartE2EDuration="2.606003168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:57.600436097 +0000 UTC m=+1462.101139823" watchObservedRunningTime="2026-10-30 19:58:57.606003168 +0000 UTC m=+1462.106706892"
Oct 30 19:58:57.859962 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:57.859821    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:58.867559 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:58.867512    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:59.578543 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.578450    4519 generic.go:356] "Generic (PLEG): container finished" podID="6881d466-0536-4777-8072-0dc9d3733166" containerID="0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c" exitCode=0
Oct 30 19:58:59.578805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.578629    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" event={"ID":"6881d466-0536-4777-8072-0dc9d3733166","Type":"ContainerDied","Data":"0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c"}
Oct 30 19:58:59.578805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.578773    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:59.579605 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.579578    4519 scope.go:122] "RemoveContainer" containerID="0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c"
Oct 30 19:58:59.579988 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.579965    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 19:58:59.581660 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.581633    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:59.581727 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.581704    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:59.581766 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:58:59.581736    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:59:00.594036 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:00.593997    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" event={"ID":"6881d466-0536-4777-8072-0dc9d3733166","Type":"ContainerStarted","Data":"f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31"}
Oct 30 19:59:01.570949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:01.570911    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:59:01.571356 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:01.571265    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:59:01.571549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:01.571519    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 19:59:02.503765 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:59:02.503694    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:59:02.579010 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:02.578961    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:59:02.579365 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:02.579327    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 19:59:03.572994 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:03.572950    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:59:03.573680 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:03.573606    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 19:59:04.575210 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:04.575166    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:59:04.576094 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:04.576056    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 19:59:05.570490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:05.569901    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:59:05.570490 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:05.569967    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 19:59:06.575558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:06.575416    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:59:06.575558 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:06.575548    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 19:59:07.574777 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:07.574739    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:59:07.575201 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:07.575172    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:59:07.575917 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:07.575897    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 19:59:14.118602 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.118566    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"]
Oct 30 19:59:14.119220 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.118783    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy" containerID="cri-o://9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e" gracePeriod=30
Oct 30 19:59:14.131790 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.131746    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"]
Oct 30 19:59:14.132058 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.132007    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy" containerID="cri-o://b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428" gracePeriod=30
Oct 30 19:59:14.144861 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.138480    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"]
Oct 30 19:59:14.144861 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.138714    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy" containerID="cri-o://838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e" gracePeriod=30
Oct 30 19:59:14.302140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.302100    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"]
Oct 30 19:59:14.302432 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.302402    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy" containerID="cri-o://156a1b4bf3fa72f380673944da2139ec03bc4002efcc19caab93003e8fe14043" gracePeriod=30
Oct 30 19:59:14.307859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.307431    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"]
Oct 30 19:59:14.307859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.307702    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy" containerID="cri-o://4f569f56280fa93a7f29a38ae77e644402bc6acbe6691e5cebc02c0172e3291e" gracePeriod=30
Oct 30 19:59:14.312598 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.312571    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"]
Oct 30 19:59:14.312801 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:14.312780    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy" containerID="cri-o://db5729905b2d45222c6aeac75c503553764aea44d6dd778bb2924466cc4fbade" gracePeriod=30
Oct 30 19:59:19.628243 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.628189    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.644750    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-kube-api-access-7nwzh\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-kube-api-access-7nwzh\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.644796    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-envoy\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.644831    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-data\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.644906    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-socket\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.645000    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-certs\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.645098    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-credential-socket\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.645150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-token\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-token\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.645238    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istiod-ca-cert\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.646055 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.645394    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-podinfo\") pod \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\" (UID: \"1a0fea40-0fee-47ae-bdef-739eeb5b8896\") "
Oct 30 19:59:19.648068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.647065    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-certs" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.648068 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.647534    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-socket" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.648443 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.648408    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-credential-socket" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.648758 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.648726    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istiod-ca-cert" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:19.649188 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.649167    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-data" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.661282 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.661204    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-token" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.661671 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.661622    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-envoy" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.671792 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.670754    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-kube-api-access-7nwzh" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "kube-api-access-7nwzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.693063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.693008    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-podinfo" pod "1a0fea40-0fee-47ae-bdef-739eeb5b8896" (UID: "1a0fea40-0fee-47ae-bdef-739eeb5b8896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:19.713356 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.713327    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:59:19.717323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.717288    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:59:19.748849 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.748807    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-kube-api-access-kbxjh\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-kube-api-access-kbxjh\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.748872    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-token\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-token\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.748916    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-envoy\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749044 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.748945    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-credential-socket\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.749221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749066    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-certs\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.749221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749113    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7bcafa-0551-4059-87db-5b466c2e914a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7bcafa-0551-4059-87db-5b466c2e914a-istiod-ca-cert\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749221 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749188    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-socket\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749233    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-certs\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749288    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-podinfo\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749380 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749353    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-socket\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.749561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749417    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-token\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-token\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.749561 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749531    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istiod-ca-cert\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.749667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749598    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-data\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749667 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749626    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-data\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.749817 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749797    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-kube-api-access-nzszb\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-kube-api-access-nzszb\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.749876 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749858    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-credential-socket\") pod \"9e7bcafa-0551-4059-87db-5b466c2e914a\" (UID: \"9e7bcafa-0551-4059-87db-5b466c2e914a\") "
Oct 30 19:59:19.749937 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.749922    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-podinfo\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.750018 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.750004    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-envoy\") pod \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\" (UID: \"f0f33475-5927-4d76-a197-4a5d0fb1aa94\") "
Oct 30 19:59:19.751445 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.751416    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.751778 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.751759    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwzh\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-kube-api-access-7nwzh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.751889 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.751876    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.751974 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.751964    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.752054 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.752044    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.752140 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.752130    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.752219 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.752209    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0fea40-0fee-47ae-bdef-739eeb5b8896-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.752299 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.752289    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.752376 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.752367    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0fea40-0fee-47ae-bdef-739eeb5b8896-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.754825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.753648    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7bcafa-0551-4059-87db-5b466c2e914a-istiod-ca-cert" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:19.754825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.753907    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-data" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.754825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.754108    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-socket" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.754825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.754401    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-certs" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.754825 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.754655    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-credential-socket" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.755449 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.755386    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-envoy" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.756556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.755999    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-socket" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.756556 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.756288    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istiod-ca-cert" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:19.756736 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.756554    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-data" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.757069 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.757042    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-credential-socket" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.757245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.757226    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-certs" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.757978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.757931    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-kube-api-access-nzszb" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "kube-api-access-nzszb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.758927 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.758907    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-podinfo" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:19.763949 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.763906    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-podinfo" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:19.767133 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.767079    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-token" pod "f0f33475-5927-4d76-a197-4a5d0fb1aa94" (UID: "f0f33475-5927-4d76-a197-4a5d0fb1aa94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.768279 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.768250    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-envoy" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.768613 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.768579    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-kube-api-access-kbxjh" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "kube-api-access-kbxjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.768844 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.768815    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-token" pod "9e7bcafa-0551-4059-87db-5b466c2e914a" (UID: "9e7bcafa-0551-4059-87db-5b466c2e914a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.854263 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.854190    4519 generic.go:356] "Generic (PLEG): container finished" podID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerID="db5729905b2d45222c6aeac75c503553764aea44d6dd778bb2924466cc4fbade" exitCode=0
Oct 30 19:59:19.854802 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.854767    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" event={"ID":"30f9d75c-5271-41f1-9777-6216ad94edf8","Type":"ContainerDied","Data":"db5729905b2d45222c6aeac75c503553764aea44d6dd778bb2924466cc4fbade"}
Oct 30 19:59:19.855773 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855745    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855780    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855795    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxjh\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-kube-api-access-kbxjh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855817    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855832    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855845    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855859    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855877    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7bcafa-0551-4059-87db-5b466c2e914a-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855889    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.855908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855902    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855916    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855934    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855948    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855963    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855976    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.855994    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f33475-5927-4d76-a197-4a5d0fb1aa94-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.856008    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzszb\" (UniqueName: \"kubernetes.io/projected/f0f33475-5927-4d76-a197-4a5d0fb1aa94-kube-api-access-nzszb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.856353 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.856023    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7bcafa-0551-4059-87db-5b466c2e914a-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.860063 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.860027    4519 generic.go:356] "Generic (PLEG): container finished" podID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerID="9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e" exitCode=0
Oct 30 19:59:19.860263 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.860210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" event={"ID":"f0f33475-5927-4d76-a197-4a5d0fb1aa94","Type":"ContainerDied","Data":"9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e"}
Oct 30 19:59:19.860323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.860186    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"
Oct 30 19:59:19.860323 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.860307    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4" event={"ID":"f0f33475-5927-4d76-a197-4a5d0fb1aa94","Type":"ContainerDied","Data":"4b4d0bca15850c72c23b15ce6d115d1d7c452d75b4e8de6b24d18d01ae0958da"}
Oct 30 19:59:19.860414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.860328    4519 scope.go:122] "RemoveContainer" containerID="9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e"
Oct 30 19:59:19.861518 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.861496    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:59:19.864745 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.864717    4519 generic.go:356] "Generic (PLEG): container finished" podID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerID="838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e" exitCode=0
Oct 30 19:59:19.865174 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.865151    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" event={"ID":"9e7bcafa-0551-4059-87db-5b466c2e914a","Type":"ContainerDied","Data":"838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e"}
Oct 30 19:59:19.865332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.865314    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv" event={"ID":"9e7bcafa-0551-4059-87db-5b466c2e914a","Type":"ContainerDied","Data":"3fadc7e91a4f84ccb4755d4846387a15b723ae1d9fa7d5ca8478d3878bd1e722"}
Oct 30 19:59:19.865540 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.865524    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"
Oct 30 19:59:19.877411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.875097    4519 generic.go:356] "Generic (PLEG): container finished" podID="af5e971a-1282-4810-a3e4-afe14b00225a" containerID="4f569f56280fa93a7f29a38ae77e644402bc6acbe6691e5cebc02c0172e3291e" exitCode=0
Oct 30 19:59:19.877411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.875201    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g" event={"ID":"af5e971a-1282-4810-a3e4-afe14b00225a","Type":"ContainerDied","Data":"4f569f56280fa93a7f29a38ae77e644402bc6acbe6691e5cebc02c0172e3291e"}
Oct 30 19:59:19.877411 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.875315    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"
Oct 30 19:59:19.884135 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.882306    4519 generic.go:356] "Generic (PLEG): container finished" podID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerID="b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428" exitCode=0
Oct 30 19:59:19.884135 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.882372    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"
Oct 30 19:59:19.884135 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.882403    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" event={"ID":"1a0fea40-0fee-47ae-bdef-739eeb5b8896","Type":"ContainerDied","Data":"b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428"}
Oct 30 19:59:19.884135 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.882454    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58" event={"ID":"1a0fea40-0fee-47ae-bdef-739eeb5b8896","Type":"ContainerDied","Data":"296038a19e243d7b08689f4bcce33b59a5e2dc16e79fcc30c23b5ff0fea5ea67"}
Oct 30 19:59:19.888902 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.888777    4519 generic.go:356] "Generic (PLEG): container finished" podID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerID="156a1b4bf3fa72f380673944da2139ec03bc4002efcc19caab93003e8fe14043" exitCode=0
Oct 30 19:59:19.888902 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.888829    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" event={"ID":"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40","Type":"ContainerDied","Data":"156a1b4bf3fa72f380673944da2139ec03bc4002efcc19caab93003e8fe14043"}
Oct 30 19:59:19.902570 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.902504    4519 scope.go:122] "RemoveContainer" containerID="9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e"
Oct 30 19:59:19.903140 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:59:19.903005    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e\": container with ID starting with 9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e not found: ID does not exist" containerID="9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e"
Oct 30 19:59:19.903348 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.903316    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e"} err="failed to get container status \"9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e\": rpc error: code = NotFound desc = could not find container \"9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e\": container with ID starting with 9d1725ec1b3401e33465531911ae8fc9007bf529a6c3b9bf40c7fce756e1ba4e not found: ID does not exist"
Oct 30 19:59:19.903913 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.903451    4519 scope.go:122] "RemoveContainer" containerID="838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e"
Oct 30 19:59:19.941908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.941879    4519 scope.go:122] "RemoveContainer" containerID="838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e"
Oct 30 19:59:19.942999 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:59:19.942970    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e\": container with ID starting with 838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e not found: ID does not exist" containerID="838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e"
Oct 30 19:59:19.943191 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.943161    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e"} err="failed to get container status \"838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e\": rpc error: code = NotFound desc = could not find container \"838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e\": container with ID starting with 838e851c98c3385075ae98a0e93f9f744aeaba190af15154964fad9452e80f2e not found: ID does not exist"
Oct 30 19:59:19.943285 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.943274    4519 scope.go:122] "RemoveContainer" containerID="4f569f56280fa93a7f29a38ae77e644402bc6acbe6691e5cebc02c0172e3291e"
Oct 30 19:59:19.959207 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959174    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-kube-api-access-mq745\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-kube-api-access-mq745\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.959381 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959270    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af5e971a-1282-4810-a3e4-afe14b00225a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e971a-1282-4810-a3e4-afe14b00225a-istio-podinfo\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.959502 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959485    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-istio-token\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-istio-token\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.959579 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959542    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-data\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.959674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959659    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-credential-socket\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.959739 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959711    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5e971a-1282-4810-a3e4-afe14b00225a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e971a-1282-4810-a3e4-afe14b00225a-istiod-ca-cert\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.959795 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959776    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-socket\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.959930 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.959901    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-certs\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.960183 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.960151    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-data" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.960377 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.960340    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-envoy\") pod \"af5e971a-1282-4810-a3e4-afe14b00225a\" (UID: \"af5e971a-1282-4810-a3e4-afe14b00225a\") "
Oct 30 19:59:19.962601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.961438    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-certs" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.962601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.961665    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-socket" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.962601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.961851    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5e971a-1282-4810-a3e4-afe14b00225a-istiod-ca-cert" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:19.962601 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.962150    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-credential-socket" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.964215 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.964184    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.964319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.964219    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.964319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.964235    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e971a-1282-4810-a3e4-afe14b00225a-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.964319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.964248    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.964319 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.964260    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:19.964933 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.964903    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af5e971a-1282-4810-a3e4-afe14b00225a-istio-podinfo" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:19.965033 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.964905    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-kube-api-access-mq745" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "kube-api-access-mq745". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.965330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.965204    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-istio-token" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:19.965330 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.965298    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-envoy" pod "af5e971a-1282-4810-a3e4-afe14b00225a" (UID: "af5e971a-1282-4810-a3e4-afe14b00225a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:19.972781 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.972746    4519 scope.go:122] "RemoveContainer" containerID="b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428"
Oct 30 19:59:19.977782 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.977641    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:59:19.993199 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.993153    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:59:19.998222 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:19.998188    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"]
Oct 30 19:59:20.006262 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.006230    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-kp5z4"]
Oct 30 19:59:20.007786 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.007759    4519 scope.go:122] "RemoveContainer" containerID="b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428"
Oct 30 19:59:20.008180 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:59:20.008149    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428\": container with ID starting with b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428 not found: ID does not exist" containerID="b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428"
Oct 30 19:59:20.008254 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.008182    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428"} err="failed to get container status \"b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428\": rpc error: code = NotFound desc = could not find container \"b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428\": container with ID starting with b0ada8f1a70b446f754ac35a377e077cd6746eef077a9956d4728eaa90357428 not found: ID does not exist"
Oct 30 19:59:20.053547 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.047816    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"]
Oct 30 19:59:20.057484 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.056027    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-v5vjv"]
Oct 30 19:59:20.064984 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.064948    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-envoy\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.065163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.064998    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-credential-socket\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.065163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.065032    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-podinfo\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.065163 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.065070    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-kube-api-access-q825f\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-kube-api-access-q825f\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.065724 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.065675    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-credential-socket" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.065873 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.065847    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-certs" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.066024 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.065145    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-certs\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.066372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.066075    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-token\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-token\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.066372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.066242    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-envoy\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.066372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.066275    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-certs\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.066372 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.066361    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-podinfo\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.067672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.066925    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-certs" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.067672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067297    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-data\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.067672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067388    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f9d75c-5271-41f1-9777-6216ad94edf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f9d75c-5271-41f1-9777-6216ad94edf8-istiod-ca-cert\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.067672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067428    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-socket\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.067672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067483    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-socket\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.067672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067544    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istiod-ca-cert\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.067672 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067642    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-data\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.068212 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067687    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-token\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-token\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.068212 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067738    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-credential-socket\") pod \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\" (UID: \"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40\") "
Oct 30 19:59:20.068212 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.067783    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-kube-api-access-8s8tg\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-kube-api-access-8s8tg\") pod \"30f9d75c-5271-41f1-9777-6216ad94edf8\" (UID: \"30f9d75c-5271-41f1-9777-6216ad94edf8\") "
Oct 30 19:59:20.068707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.068682    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.068871 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.068857    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.071567    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.071593    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e971a-1282-4810-a3e4-afe14b00225a-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.071609    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq745\" (UniqueName: \"kubernetes.io/projected/af5e971a-1282-4810-a3e4-afe14b00225a-kube-api-access-mq745\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.071632    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.071648    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e971a-1282-4810-a3e4-afe14b00225a-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.068700    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-podinfo" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.069983    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istiod-ca-cert" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.070206    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-data" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.070262    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-podinfo" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.070550    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-socket" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.070591    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f9d75c-5271-41f1-9777-6216ad94edf8-istiod-ca-cert" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.070748    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-socket" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.071928    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-credential-socket" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.073707 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.072449    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-data" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.074530 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.074196    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"]
Oct 30 19:59:20.074530 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.074502    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-kube-api-access-8s8tg" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "kube-api-access-8s8tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:20.074880 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.074858    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-envoy" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.074958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.074876    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-kube-api-access-q825f" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "kube-api-access-q825f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:20.074958 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.074902    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-token" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:20.075062 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.074965    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-envoy" pod "30f9d75c-5271-41f1-9777-6216ad94edf8" (UID: "30f9d75c-5271-41f1-9777-6216ad94edf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:20.076255 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.076229    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-token" pod "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" (UID: "808e5138-e9a8-4ccf-97e0-a9ddd57bfb40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:20.079218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.079188    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-drx58"]
Oct 30 19:59:20.173303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173259    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173303 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173303    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173320    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173333    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f9d75c-5271-41f1-9777-6216ad94edf8-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173347    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173359    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173378    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173391    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173403    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173415    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173430    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8tg\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-kube-api-access-8s8tg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173444    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173458    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173498    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q825f\" (UniqueName: \"kubernetes.io/projected/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40-kube-api-access-q825f\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.173611 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.173513    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f9d75c-5271-41f1-9777-6216ad94edf8-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:20.304537 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.304494    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"]
Oct 30 19:59:20.309272 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.309232    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-c4j8g"]
Oct 30 19:59:20.930797 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.930766    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"
Oct 30 19:59:20.931423 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.930785    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9" event={"ID":"30f9d75c-5271-41f1-9777-6216ad94edf8","Type":"ContainerDied","Data":"385ed133d071f075fb26f6d472e69be0307257797fd541e2817bef41e65dbb75"}
Oct 30 19:59:20.931675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.931486    4519 scope.go:122] "RemoveContainer" containerID="db5729905b2d45222c6aeac75c503553764aea44d6dd778bb2924466cc4fbade"
Oct 30 19:59:20.941669 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.941630    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf" event={"ID":"808e5138-e9a8-4ccf-97e0-a9ddd57bfb40","Type":"ContainerDied","Data":"e735576afbf2789f1c658e19f012e6fcdcc79e84f70e4eb6dbb5c2609af7f9aa"}
Oct 30 19:59:20.941846 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.941765    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"
Oct 30 19:59:20.962722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:20.956695    4519 scope.go:122] "RemoveContainer" containerID="156a1b4bf3fa72f380673944da2139ec03bc4002efcc19caab93003e8fe14043"
Oct 30 19:59:21.022153 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.022110    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"]
Oct 30 19:59:21.029732 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.029692    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-t6vlf"]
Oct 30 19:59:21.051866 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.051825    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"]
Oct 30 19:59:21.059072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.059019    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-d42h9"]
Oct 30 19:59:21.625534 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.625493    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" path="/var/lib/kubelet/pods/1a0fea40-0fee-47ae-bdef-739eeb5b8896/volumes"
Oct 30 19:59:21.626155 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.626130    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" path="/var/lib/kubelet/pods/30f9d75c-5271-41f1-9777-6216ad94edf8/volumes"
Oct 30 19:59:21.626748 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.626727    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" path="/var/lib/kubelet/pods/808e5138-e9a8-4ccf-97e0-a9ddd57bfb40/volumes"
Oct 30 19:59:21.627640 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.627610    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" path="/var/lib/kubelet/pods/9e7bcafa-0551-4059-87db-5b466c2e914a/volumes"
Oct 30 19:59:21.628410 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.628386    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" path="/var/lib/kubelet/pods/af5e971a-1282-4810-a3e4-afe14b00225a/volumes"
Oct 30 19:59:21.629074 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:21.629055    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" path="/var/lib/kubelet/pods/f0f33475-5927-4d76-a197-4a5d0fb1aa94/volumes"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.119151    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"]
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120200    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120233    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120260    4519 cpu_manager.go:414] "containerMap: removing container" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120269    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120315    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120324    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120358    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120366    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120383    4519 cpu_manager.go:414] "containerMap: removing container" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120393    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120415    4519 cpu_manager.go:414] "containerMap: removing container" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120426    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120672    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0fea40-0fee-47ae-bdef-739eeb5b8896" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120699    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="808e5138-e9a8-4ccf-97e0-a9ddd57bfb40" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120714    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f9d75c-5271-41f1-9777-6216ad94edf8" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120731    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7bcafa-0551-4059-87db-5b466c2e914a" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120745    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5e971a-1282-4810-a3e4-afe14b00225a" containerName="istio-proxy"
Oct 30 19:59:26.125284 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.120761    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f33475-5927-4d76-a197-4a5d0fb1aa94" containerName="istio-proxy"
Oct 30 19:59:26.137878 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.137843    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.299223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.210347    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:26.299223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.210435    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:59:26.299223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.262071    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.299223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.262167    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.299223 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.262548    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rm45\" (UniqueName: \"kubernetes.io/projected/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-kube-api-access-5rm45\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.374674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.373510    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rm45\" (UniqueName: \"kubernetes.io/projected/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-kube-api-access-5rm45\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.374674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.373809    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.374674 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.373871    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.376278 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.376037    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.376278 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.376113    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"]
Oct 30 19:59:26.396498 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.396442    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.443001 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.440650    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rm45\" (UniqueName: \"kubernetes.io/projected/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-kube-api-access-5rm45\") pod \"maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.667386 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.667347    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:26.996715 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:59:26.996676    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1320673f_eaec_48e3_8bb1_3ad0c9e2a94e.slice/crio-a40b3b9e4eb7c7c1214f0f7c0ea545fcac7df1b5aa2cdaa6faf4495d950db039 WatchSource:0}: Error finding container a40b3b9e4eb7c7c1214f0f7c0ea545fcac7df1b5aa2cdaa6faf4495d950db039: Status 404 returned error can't find the container with id a40b3b9e4eb7c7c1214f0f7c0ea545fcac7df1b5aa2cdaa6faf4495d950db039
Oct 30 19:59:26.997239 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:26.997211    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"]
Oct 30 19:59:27.169805 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:27.169757    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr" event={"ID":"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e","Type":"ContainerStarted","Data":"a40b3b9e4eb7c7c1214f0f7c0ea545fcac7df1b5aa2cdaa6faf4495d950db039"}
Oct 30 19:59:28.187046 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:28.187000    4519 generic.go:356] "Generic (PLEG): container finished" podID="1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" containerID="d1c9cd48fbfbf1c726f3581d9ace569a070fe3be4844d3fb8f223438c42e8f90" exitCode=0
Oct 30 19:59:28.187570 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:28.187088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr" event={"ID":"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e","Type":"ContainerDied","Data":"d1c9cd48fbfbf1c726f3581d9ace569a070fe3be4844d3fb8f223438c42e8f90"}
Oct 30 19:59:29.381978 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.381943    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:29.435729 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.435688    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-openshift-ca\") pod \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") "
Oct 30 19:59:29.435939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.435839    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-xks-ca\") pod \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") "
Oct 30 19:59:29.435939 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.435878    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-kube-api-access-5rm45\" (UniqueName: \"kubernetes.io/projected/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-kube-api-access-5rm45\") pod \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\" (UID: \"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e\") "
Oct 30 19:59:29.436549 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.436512    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-openshift-ca" pod "1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" (UID: "1320673f-eaec-48e3-8bb1-3ad0c9e2a94e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:29.436786 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.436765    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:29.439419 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.439378    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-xks-ca" pod "1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" (UID: "1320673f-eaec-48e3-8bb1-3ad0c9e2a94e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:29.439711 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.439680    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-kube-api-access-5rm45" pod "1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" (UID: "1320673f-eaec-48e3-8bb1-3ad0c9e2a94e"). InnerVolumeSpecName "kube-api-access-5rm45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:29.542501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.540998    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:29.542501 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:29.541047    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rm45\" (UniqueName: \"kubernetes.io/projected/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e-kube-api-access-5rm45\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:30.208202 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:30.208170    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"
Oct 30 19:59:30.208401 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:30.208202    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr" event={"ID":"1320673f-eaec-48e3-8bb1-3ad0c9e2a94e","Type":"ContainerDied","Data":"a40b3b9e4eb7c7c1214f0f7c0ea545fcac7df1b5aa2cdaa6faf4495d950db039"}
Oct 30 19:59:30.208401 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:30.208281    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40b3b9e4eb7c7c1214f0f7c0ea545fcac7df1b5aa2cdaa6faf4495d950db039"
Oct 30 19:59:32.234933 ip-10-0-133-26 kubenswrapper[4519]: E1030 19:59:32.234888    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 19:59:41.032048 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.032003    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"]
Oct 30 19:59:41.036685 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.036639    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8308e4b9-d5862"]
Oct 30 19:59:41.291152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.291068    4519 scope.go:122] "RemoveContainer" containerID="cbfff8ffb3366f4977f6cb4871ddae598329e0d143e97533ab0b9f47b4a27c04"
Oct 30 19:59:41.302767 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.302740    4519 scope.go:122] "RemoveContainer" containerID="d9dc91dc8dc01e9fa0d91a26dec4e2cad736f07322dadf40af9cad8a2a91fbca"
Oct 30 19:59:41.314132 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.314106    4519 scope.go:122] "RemoveContainer" containerID="5502876e5be5137c8ce734d7dffa951e7895babdd956ad46fb43d5d0d40a736b"
Oct 30 19:59:41.325394 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.325363    4519 scope.go:122] "RemoveContainer" containerID="1aa1e4c73cf32d9c7b8b823f10c999be50ca2269cbc29e84784b935ff9bda361"
Oct 30 19:59:41.338407 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.338173    4519 scope.go:122] "RemoveContainer" containerID="b419a095d132e6a2b256b70169e09481e0119531a3712c81517ea8a72ebfdfc0"
Oct 30 19:59:41.354434 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.354403    4519 scope.go:122] "RemoveContainer" containerID="a04bb3d251bc055230d4fbed62863dfc7de075e941e468ae29251e7463b54de1"
Oct 30 19:59:41.370382 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.370350    4519 scope.go:122] "RemoveContainer" containerID="1c8ca4f232c02c76b22f1c8310e8047322976b3258c9ab93ced01c6c60ff8602"
Oct 30 19:59:41.385049 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.385016    4519 scope.go:122] "RemoveContainer" containerID="7ead5d6ae5a54989b4046623efd5955cb9c3881bbc5c9e80cc1a77b7d27603ca"
Oct 30 19:59:41.400181 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.400152    4519 scope.go:122] "RemoveContainer" containerID="1ba42a64acd05a7170ba3c377567cf2f8cfc0d5c1556d921b8dc9a1cd0eefe29"
Oct 30 19:59:41.412245 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.412216    4519 scope.go:122] "RemoveContainer" containerID="cbc52c82f31d3dfd1831d645daed18c80cb7ebcf60e02f2c29b54d3ef3869c25"
Oct 30 19:59:41.423238 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.423214    4519 scope.go:122] "RemoveContainer" containerID="95784d5b194fa9d954b95c4af63dfa6b9aef51a83e601da834a251e319e0ed41"
Oct 30 19:59:41.434092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.434065    4519 scope.go:122] "RemoveContainer" containerID="6942d6709587b4f75dad4e823c5ec093c82fdfd8387cd57995b671d8fdb8c834"
Oct 30 19:59:41.625675 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:41.625632    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd820d2-d4d4-4002-8843-1136087a44ab" path="/var/lib/kubelet/pods/dbd820d2-d4d4-4002-8843-1136087a44ab/volumes"
Oct 30 19:59:44.400440 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.400402    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"]
Oct 30 19:59:44.401092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.401076    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" containerName="revoke-keys"
Oct 30 19:59:44.401092 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.401092    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" containerName="revoke-keys"
Oct 30 19:59:44.401218 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.401209    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" containerName="revoke-keys"
Oct 30 19:59:44.408622 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.408594    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.425072 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.425035    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"]
Oct 30 19:59:44.458000 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.457958    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 19:59:44.458680 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.458651    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:44.528653 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.528596    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.528859 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.528805    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.528917 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.528868    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4jv\" (UniqueName: \"kubernetes.io/projected/9f71194d-b774-4e3c-9dfa-1aeef49413fd-kube-api-access-cs4jv\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.630670 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.630624    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.630908 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.630711    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4jv\" (UniqueName: \"kubernetes.io/projected/9f71194d-b774-4e3c-9dfa-1aeef49413fd-kube-api-access-cs4jv\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.630974 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.630954    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.631324 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.631302    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.633377 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.633352    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.651225 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.651144    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4jv\" (UniqueName: \"kubernetes.io/projected/9f71194d-b774-4e3c-9dfa-1aeef49413fd-kube-api-access-cs4jv\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-cxbg2\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:44.724979 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:44.724934    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:45.044515 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:59:45.044461    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f71194d_b774_4e3c_9dfa_1aeef49413fd.slice/crio-e6b3197e10f250e6a6da5164ab008167e57f8a6135bd7b7644730543743bf340 WatchSource:0}: Error finding container e6b3197e10f250e6a6da5164ab008167e57f8a6135bd7b7644730543743bf340: Status 404 returned error can't find the container with id e6b3197e10f250e6a6da5164ab008167e57f8a6135bd7b7644730543743bf340
Oct 30 19:59:45.045969 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.045941    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"]
Oct 30 19:59:45.267065 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.267028    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"]
Oct 30 19:59:45.272636 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.272602    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.294383 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.294303    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"]
Oct 30 19:59:45.412563 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.412528    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2" event={"ID":"9f71194d-b774-4e3c-9dfa-1aeef49413fd","Type":"ContainerStarted","Data":"e6b3197e10f250e6a6da5164ab008167e57f8a6135bd7b7644730543743bf340"}
Oct 30 19:59:45.444546 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.444456    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.444722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.444555    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jkmh\" (UniqueName: \"kubernetes.io/projected/355594ec-cf9f-4148-bca2-4313dfc25b89-kube-api-access-7jkmh\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.444722 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.444677    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.546869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.546763    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.546869 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.546813    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jkmh\" (UniqueName: \"kubernetes.io/projected/355594ec-cf9f-4148-bca2-4313dfc25b89-kube-api-access-7jkmh\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.547121 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.546921    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.547864 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.547833    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.549609 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.549581    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.567809 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.567770    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jkmh\" (UniqueName: \"kubernetes.io/projected/355594ec-cf9f-4148-bca2-4313dfc25b89-kube-api-access-7jkmh\") pod \"maas-api-revoke-keys-e2e-api-a-93263c-558cb\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.591679 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.591637    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:45.979526 ip-10-0-133-26 kubenswrapper[4519]: W1030 19:59:45.978235    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355594ec_cf9f_4148_bca2_4313dfc25b89.slice/crio-f002c3f1f057462d29c10d23b639677c37ad4427ef5d7615012d0af79c81dd8c WatchSource:0}: Error finding container f002c3f1f057462d29c10d23b639677c37ad4427ef5d7615012d0af79c81dd8c: Status 404 returned error can't find the container with id f002c3f1f057462d29c10d23b639677c37ad4427ef5d7615012d0af79c81dd8c
Oct 30 19:59:45.979526 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:45.979207    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"]
Oct 30 19:59:46.422359 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:46.422316    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb" event={"ID":"355594ec-cf9f-4148-bca2-4313dfc25b89","Type":"ContainerStarted","Data":"b2396c7df785453f210a988a2abe6c802f3f65bb206e759edd647675e2cb7f21"}
Oct 30 19:59:46.422903 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:46.422383    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb" event={"ID":"355594ec-cf9f-4148-bca2-4313dfc25b89","Type":"ContainerStarted","Data":"f002c3f1f057462d29c10d23b639677c37ad4427ef5d7615012d0af79c81dd8c"}
Oct 30 19:59:46.424105 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:46.424074    4519 generic.go:356] "Generic (PLEG): container finished" podID="9f71194d-b774-4e3c-9dfa-1aeef49413fd" containerID="f5ec55756893f30542d8ec4773ece8f3c1873944615dba3ffe127c8fa79178d3" exitCode=0
Oct 30 19:59:46.424240 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:46.424162    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2" event={"ID":"9f71194d-b774-4e3c-9dfa-1aeef49413fd","Type":"ContainerDied","Data":"f5ec55756893f30542d8ec4773ece8f3c1873944615dba3ffe127c8fa79178d3"}
Oct 30 19:59:46.463918 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:46.463860    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb" podStartSLOduration=1.463842995 podStartE2EDuration="1.463842995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:46.462074662 +0000 UTC m=+1510.962778386" watchObservedRunningTime="2026-10-30 19:59:46.463842995 +0000 UTC m=+1510.964546719"
Oct 30 19:59:47.435688 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.435645    4519 generic.go:356] "Generic (PLEG): container finished" podID="355594ec-cf9f-4148-bca2-4313dfc25b89" containerID="b2396c7df785453f210a988a2abe6c802f3f65bb206e759edd647675e2cb7f21" exitCode=0
Oct 30 19:59:47.436136 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.435737    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb" event={"ID":"355594ec-cf9f-4148-bca2-4313dfc25b89","Type":"ContainerDied","Data":"b2396c7df785453f210a988a2abe6c802f3f65bb206e759edd647675e2cb7f21"}
Oct 30 19:59:47.726414 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.726381    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:47.800037 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.799994    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-xks-ca\") pod \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") "
Oct 30 19:59:47.800264 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.800157    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-openshift-ca\") pod \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") "
Oct 30 19:59:47.800346 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.800269    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f71194d-b774-4e3c-9dfa-1aeef49413fd-kube-api-access-cs4jv\" (UniqueName: \"kubernetes.io/projected/9f71194d-b774-4e3c-9dfa-1aeef49413fd-kube-api-access-cs4jv\") pod \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\" (UID: \"9f71194d-b774-4e3c-9dfa-1aeef49413fd\") "
Oct 30 19:59:47.801189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.800847    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-openshift-ca" pod "9f71194d-b774-4e3c-9dfa-1aeef49413fd" (UID: "9f71194d-b774-4e3c-9dfa-1aeef49413fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:47.801189 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.801135    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:47.803102 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.803067    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f71194d-b774-4e3c-9dfa-1aeef49413fd-kube-api-access-cs4jv" pod "9f71194d-b774-4e3c-9dfa-1aeef49413fd" (UID: "9f71194d-b774-4e3c-9dfa-1aeef49413fd"). InnerVolumeSpecName "kube-api-access-cs4jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:47.803294 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.803265    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-xks-ca" pod "9f71194d-b774-4e3c-9dfa-1aeef49413fd" (UID: "9f71194d-b774-4e3c-9dfa-1aeef49413fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:47.902504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.902450    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4jv\" (UniqueName: \"kubernetes.io/projected/9f71194d-b774-4e3c-9dfa-1aeef49413fd-kube-api-access-cs4jv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:47.902504 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:47.902501    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f71194d-b774-4e3c-9dfa-1aeef49413fd-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:48.445152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.445117    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"
Oct 30 19:59:48.445152 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.445146    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2" event={"ID":"9f71194d-b774-4e3c-9dfa-1aeef49413fd","Type":"ContainerDied","Data":"e6b3197e10f250e6a6da5164ab008167e57f8a6135bd7b7644730543743bf340"}
Oct 30 19:59:48.445738 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.445198    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b3197e10f250e6a6da5164ab008167e57f8a6135bd7b7644730543743bf340"
Oct 30 19:59:48.629629 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.629592    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:48.715023 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.714918    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-openshift-ca\") pod \"355594ec-cf9f-4148-bca2-4313dfc25b89\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") "
Oct 30 19:59:48.715217 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.715039    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-xks-ca\") pod \"355594ec-cf9f-4148-bca2-4313dfc25b89\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") "
Oct 30 19:59:48.715312 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.715287    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355594ec-cf9f-4148-bca2-4313dfc25b89-kube-api-access-7jkmh\" (UniqueName: \"kubernetes.io/projected/355594ec-cf9f-4148-bca2-4313dfc25b89-kube-api-access-7jkmh\") pod \"355594ec-cf9f-4148-bca2-4313dfc25b89\" (UID: \"355594ec-cf9f-4148-bca2-4313dfc25b89\") "
Oct 30 19:59:48.715407 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.715378    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-openshift-ca" pod "355594ec-cf9f-4148-bca2-4313dfc25b89" (UID: "355594ec-cf9f-4148-bca2-4313dfc25b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:48.716596 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.716568    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:48.718332 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.718290    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355594ec-cf9f-4148-bca2-4313dfc25b89-kube-api-access-7jkmh" pod "355594ec-cf9f-4148-bca2-4313dfc25b89" (UID: "355594ec-cf9f-4148-bca2-4313dfc25b89"). InnerVolumeSpecName "kube-api-access-7jkmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:48.726196 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.726145    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-xks-ca" pod "355594ec-cf9f-4148-bca2-4313dfc25b89" (UID: "355594ec-cf9f-4148-bca2-4313dfc25b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:48.818655 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.818614    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355594ec-cf9f-4148-bca2-4313dfc25b89-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:48.818655 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:48.818651    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jkmh\" (UniqueName: \"kubernetes.io/projected/355594ec-cf9f-4148-bca2-4313dfc25b89-kube-api-access-7jkmh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 19:59:49.469489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:49.468031    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"
Oct 30 19:59:49.469489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:49.468034    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb" event={"ID":"355594ec-cf9f-4148-bca2-4313dfc25b89","Type":"ContainerDied","Data":"f002c3f1f057462d29c10d23b639677c37ad4427ef5d7615012d0af79c81dd8c"}
Oct 30 19:59:49.469489 ip-10-0-133-26 kubenswrapper[4519]: I1030 19:59:49.468175    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f002c3f1f057462d29c10d23b639677c37ad4427ef5d7615012d0af79c81dd8c"
Oct 30 20:00:00.154406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.154365    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"]
Oct 30 20:00:00.155104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.155084    4519 cpu_manager.go:414] "containerMap: removing container" podUID="355594ec-cf9f-4148-bca2-4313dfc25b89" containerName="revoke-keys"
Oct 30 20:00:00.155150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.155106    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355594ec-cf9f-4148-bca2-4313dfc25b89" containerName="revoke-keys"
Oct 30 20:00:00.155150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.155127    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9f71194d-b774-4e3c-9dfa-1aeef49413fd" containerName="revoke-keys"
Oct 30 20:00:00.155150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.155133    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f71194d-b774-4e3c-9dfa-1aeef49413fd" containerName="revoke-keys"
Oct 30 20:00:00.155244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.155225    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="355594ec-cf9f-4148-bca2-4313dfc25b89" containerName="revoke-keys"
Oct 30 20:00:00.155244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.155238    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f71194d-b774-4e3c-9dfa-1aeef49413fd" containerName="revoke-keys"
Oct 30 20:00:00.159222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.159193    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.166910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.166876    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"]
Oct 30 20:00:00.172042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.171983    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.173866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.173839    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"]
Oct 30 20:00:00.179731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.179700    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"]
Oct 30 20:00:00.179912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.179821    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.185520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.185110    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"]
Oct 30 20:00:00.187615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.187567    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.190153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.190130    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:00:00.191706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.191673    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.195126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.195098    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"]
Oct 30 20:00:00.205197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.203768    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"]
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.218861    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwh6\" (UniqueName: \"kubernetes.io/projected/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-kube-api-access-7rwh6\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.219038    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.219085    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2324b1-81c5-46d8-adec-73e2aebbe111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.219278    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6h6\" (UniqueName: \"kubernetes.io/projected/07ec8783-2c39-4c35-836a-89a002783adb-kube-api-access-vb6h6\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.219582    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ec8783-2c39-4c35-836a-89a002783adb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.219634    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc26\" (UniqueName: \"kubernetes.io/projected/a278a416-54cd-4256-ac17-6e23b31c0671-kube-api-access-7cc26\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.219691    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a278a416-54cd-4256-ac17-6e23b31c0671-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.219820    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qz4\" (UniqueName: \"kubernetes.io/projected/2d2324b1-81c5-46d8-adec-73e2aebbe111-kube-api-access-z2qz4\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.226716    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.273946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.243022    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"]
Oct 30 20:00:00.308680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.306494    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"]
Oct 30 20:00:00.308680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.306563    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"]
Oct 30 20:00:00.308680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.306599    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"]
Oct 30 20:00:00.308680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.306635    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"]
Oct 30 20:00:00.308680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.306665    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"]
Oct 30 20:00:00.308680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.306873    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.324845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.324806    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"]
Oct 30 20:00:00.325125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.325097    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.333082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.333048    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"]
Oct 30 20:00:00.334958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.334934    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.336476 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336424    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftg6v\" (UniqueName: \"kubernetes.io/projected/61dc526f-7dee-49ac-b389-b340ceab3266-kube-api-access-ftg6v\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.336611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336513    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc526f-7dee-49ac-b389-b340ceab3266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.336611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336567    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwh6\" (UniqueName: \"kubernetes.io/projected/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-kube-api-access-7rwh6\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.336739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336678    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4gj\" (UniqueName: \"kubernetes.io/projected/33b31c7a-ff39-4203-9e69-e8b773b009e4-kube-api-access-nt4gj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.336739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336711    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.336848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336739    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2324b1-81c5-46d8-adec-73e2aebbe111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.336848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336796    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b31c7a-ff39-4203-9e69-e8b773b009e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.336959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336882    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f4700-7e26-4304-be69-bd6c77566780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.337011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.336957    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6h6\" (UniqueName: \"kubernetes.io/projected/07ec8783-2c39-4c35-836a-89a002783adb-kube-api-access-vb6h6\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.337067 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337027    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ec8783-2c39-4c35-836a-89a002783adb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.337122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337062    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc26\" (UniqueName: \"kubernetes.io/projected/a278a416-54cd-4256-ac17-6e23b31c0671-kube-api-access-7cc26\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.337122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337109    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a278a416-54cd-4256-ac17-6e23b31c0671-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.337219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337141    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5szkt\" (UniqueName: \"kubernetes.io/projected/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-kube-api-access-5szkt\") pod \"maas-api-key-cleanup-29889840-bbbl2\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.337219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337209    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsd69\" (UniqueName: \"kubernetes.io/projected/906f4700-7e26-4304-be69-bd6c77566780-kube-api-access-nsd69\") pod \"maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.338225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337277    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qz4\" (UniqueName: \"kubernetes.io/projected/2d2324b1-81c5-46d8-adec-73e2aebbe111-kube-api-access-z2qz4\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.338225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337310    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-bbbl2\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.338225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.337867    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2324b1-81c5-46d8-adec-73e2aebbe111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.340519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.340491    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ec8783-2c39-4c35-836a-89a002783adb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.341533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.341254    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.341758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.341455    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.350233 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.350202    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a278a416-54cd-4256-ac17-6e23b31c0671-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.357901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.357866    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"]
Oct 30 20:00:00.377418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.371718    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"]
Oct 30 20:00:00.386079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.386052    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"]
Oct 30 20:00:00.386274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.386259    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"]
Oct 30 20:00:00.386492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.386458    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.402659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.394898    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"]
Oct 30 20:00:00.402659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.398871    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qz4\" (UniqueName: \"kubernetes.io/projected/2d2324b1-81c5-46d8-adec-73e2aebbe111-kube-api-access-z2qz4\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.402659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.399537    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwh6\" (UniqueName: \"kubernetes.io/projected/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-kube-api-access-7rwh6\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.402659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.402320    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6h6\" (UniqueName: \"kubernetes.io/projected/07ec8783-2c39-4c35-836a-89a002783adb-kube-api-access-vb6h6\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.403871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.403838    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc26\" (UniqueName: \"kubernetes.io/projected/a278a416-54cd-4256-ac17-6e23b31c0671-kube-api-access-7cc26\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.406674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.406602    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"]
Oct 30 20:00:00.409226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.409202    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.412856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.412705    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.413132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.413067    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"]
Oct 30 20:00:00.421637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.421529    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"]
Oct 30 20:00:00.429183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.428794    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"]
Oct 30 20:00:00.435383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.435355    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.436292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.436262    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"]
Oct 30 20:00:00.438861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.438829    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4gj\" (UniqueName: \"kubernetes.io/projected/33b31c7a-ff39-4203-9e69-e8b773b009e4-kube-api-access-nt4gj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.439071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439044    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsgs\" (UniqueName: \"kubernetes.io/projected/64d4d50d-febd-43fe-90dc-cb8836687123-kube-api-access-2vsgs\") pod \"maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.439240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439220    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b31c7a-ff39-4203-9e69-e8b773b009e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.439354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439339    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7g4\" (UniqueName: \"kubernetes.io/projected/32cb5bde-e08d-496d-a83b-365d19d8d4a4-kube-api-access-sk7g4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.439455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439444    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rrf\" (UniqueName: \"kubernetes.io/projected/032754c4-d986-4767-bc09-19ff1ba923a7-kube-api-access-k9rrf\") pod \"maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.439576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439563    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32248e1e-4b52-4f73-b987-b0e348fab854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.439661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439651    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f4700-7e26-4304-be69-bd6c77566780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.439772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439762    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cb5bde-e08d-496d-a83b-365d19d8d4a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.439848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439838    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5sx\" (UniqueName: \"kubernetes.io/projected/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-kube-api-access-8m5sx\") pod \"maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.439921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439912    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5szkt\" (UniqueName: \"kubernetes.io/projected/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-kube-api-access-5szkt\") pod \"maas-api-key-cleanup-29889840-bbbl2\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.440008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.439999    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsd69\" (UniqueName: \"kubernetes.io/projected/906f4700-7e26-4304-be69-bd6c77566780-kube-api-access-nsd69\") pod \"maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.440083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440057    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b31c7a-ff39-4203-9e69-e8b773b009e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.440127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440072    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef72380-91f7-4631-8100-4454165123fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.440212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440202    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-bbbl2\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.440278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440270    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftg6v\" (UniqueName: \"kubernetes.io/projected/61dc526f-7dee-49ac-b389-b340ceab3266-kube-api-access-ftg6v\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.440350 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440339    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc526f-7dee-49ac-b389-b340ceab3266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.440423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440415    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032754c4-d986-4767-bc09-19ff1ba923a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.440508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440497    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57zsk\" (UniqueName: \"kubernetes.io/projected/fef72380-91f7-4631-8100-4454165123fc-kube-api-access-57zsk\") pod \"maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.440599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440589    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrpq\" (UniqueName: \"kubernetes.io/projected/32248e1e-4b52-4f73-b987-b0e348fab854-kube-api-access-zmrpq\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.440667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440658    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.440746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440737    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d4d50d-febd-43fe-90dc-cb8836687123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.441002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.440978    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-bbbl2\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.441389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.441352    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f4700-7e26-4304-be69-bd6c77566780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.442343 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.442295    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc526f-7dee-49ac-b389-b340ceab3266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.443949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.443908    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"]
Oct 30 20:00:00.449692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.449668    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.452339 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.452261    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"]
Oct 30 20:00:00.461132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.461102    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"]
Oct 30 20:00:00.467742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.466926    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:00.474911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.474872    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4gj\" (UniqueName: \"kubernetes.io/projected/33b31c7a-ff39-4203-9e69-e8b773b009e4-kube-api-access-nt4gj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.475080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.474936    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftg6v\" (UniqueName: \"kubernetes.io/projected/61dc526f-7dee-49ac-b389-b340ceab3266-kube-api-access-ftg6v\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.477385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.477354    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsd69\" (UniqueName: \"kubernetes.io/projected/906f4700-7e26-4304-be69-bd6c77566780-kube-api-access-nsd69\") pod \"maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.478527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.478505    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"]
Oct 30 20:00:00.482560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.482529    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:00.486554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.486432    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"]
Oct 30 20:00:00.492843 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.492808    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5szkt\" (UniqueName: \"kubernetes.io/projected/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-kube-api-access-5szkt\") pod \"maas-api-key-cleanup-29889840-bbbl2\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.494108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.494081    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"]
Oct 30 20:00:00.501966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.501933    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"]
Oct 30 20:00:00.511935 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.510412    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:00.542226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542167    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgs\" (UniqueName: \"kubernetes.io/projected/64d4d50d-febd-43fe-90dc-cb8836687123-kube-api-access-2vsgs\") pod \"maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.542226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542230    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwwz\" (UniqueName: \"kubernetes.io/projected/e6456596-13e7-48e2-af8b-1ec739ca9871-kube-api-access-shwwz\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:00.542516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542279    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7g4\" (UniqueName: \"kubernetes.io/projected/32cb5bde-e08d-496d-a83b-365d19d8d4a4-kube-api-access-sk7g4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.542516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542335    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbtf\" (UniqueName: \"kubernetes.io/projected/c60757fe-86c3-4d66-a25c-e1ce2a276453-kube-api-access-5sbtf\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.542758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542736    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rrf\" (UniqueName: \"kubernetes.io/projected/032754c4-d986-4767-bc09-19ff1ba923a7-kube-api-access-k9rrf\") pod \"maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.542921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542775    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32248e1e-4b52-4f73-b987-b0e348fab854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.542921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542805    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60757fe-86c3-4d66-a25c-e1ce2a276453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.542921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542927    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cb5bde-e08d-496d-a83b-365d19d8d4a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.543127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.542974    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6456596-13e7-48e2-af8b-1ec739ca9871-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:00.543127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.543013    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5sx\" (UniqueName: \"kubernetes.io/projected/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-kube-api-access-8m5sx\") pod \"maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.543127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.543124    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef72380-91f7-4631-8100-4454165123fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.543297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.543217    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032754c4-d986-4767-bc09-19ff1ba923a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.543374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.543355    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57zsk\" (UniqueName: \"kubernetes.io/projected/fef72380-91f7-4631-8100-4454165123fc-kube-api-access-57zsk\") pod \"maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.543955    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32248e1e-4b52-4f73-b987-b0e348fab854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.544034    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032754c4-d986-4767-bc09-19ff1ba923a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.544158    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cb5bde-e08d-496d-a83b-365d19d8d4a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.544287    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrpq\" (UniqueName: \"kubernetes.io/projected/32248e1e-4b52-4f73-b987-b0e348fab854-kube-api-access-zmrpq\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.544334    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.544383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d4d50d-febd-43fe-90dc-cb8836687123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.544978    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef72380-91f7-4631-8100-4454165123fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.545271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.545208    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d4d50d-febd-43fe-90dc-cb8836687123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.545801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.545403    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.549872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.549841    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:00.566698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.563921    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:00.604386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.603104    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rrf\" (UniqueName: \"kubernetes.io/projected/032754c4-d986-4767-bc09-19ff1ba923a7-kube-api-access-k9rrf\") pod \"maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.605996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.604986    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57zsk\" (UniqueName: \"kubernetes.io/projected/fef72380-91f7-4631-8100-4454165123fc-kube-api-access-57zsk\") pod \"maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.605996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.605575    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7g4\" (UniqueName: \"kubernetes.io/projected/32cb5bde-e08d-496d-a83b-365d19d8d4a4-kube-api-access-sk7g4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.605996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.605665    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrpq\" (UniqueName: \"kubernetes.io/projected/32248e1e-4b52-4f73-b987-b0e348fab854-kube-api-access-zmrpq\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.605996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.605792    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:00.621777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.608844    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:00.621777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.609114    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5sx\" (UniqueName: \"kubernetes.io/projected/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-kube-api-access-8m5sx\") pod \"maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.631075 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.630366    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsgs\" (UniqueName: \"kubernetes.io/projected/64d4d50d-febd-43fe-90dc-cb8836687123-kube-api-access-2vsgs\") pod \"maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.641518 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.640126    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:00.671804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.668518    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwwz\" (UniqueName: \"kubernetes.io/projected/e6456596-13e7-48e2-af8b-1ec739ca9871-kube-api-access-shwwz\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:00.671804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.668640    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbtf\" (UniqueName: \"kubernetes.io/projected/c60757fe-86c3-4d66-a25c-e1ce2a276453-kube-api-access-5sbtf\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.671804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.668703    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60757fe-86c3-4d66-a25c-e1ce2a276453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.671804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.668855    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6456596-13e7-48e2-af8b-1ec739ca9871-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:00.671804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.671482    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6456596-13e7-48e2-af8b-1ec739ca9871-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:00.672789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.672754    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60757fe-86c3-4d66-a25c-e1ce2a276453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.696539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.696458    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:00.707223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.706052    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:00.712218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.712137    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:00.722848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.722819    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbtf\" (UniqueName: \"kubernetes.io/projected/c60757fe-86c3-4d66-a25c-e1ce2a276453-kube-api-access-5sbtf\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.761359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.758993    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwwz\" (UniqueName: \"kubernetes.io/projected/e6456596-13e7-48e2-af8b-1ec739ca9871-kube-api-access-shwwz\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:00.762881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.762746    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:00.768614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.764750    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:00.788148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.788115    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:00.872635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.854683    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:00.872635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.860845    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:00.872635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.862132    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"]
Oct 30 20:00:00.898623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:00.898091    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:01.110929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.110826    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"]
Oct 30 20:00:01.436817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.436414    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"]
Oct 30 20:00:01.441962 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:01.441926    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61dc526f_7dee_49ac_b389_b340ceab3266.slice/crio-585628331ebab008c7bbdb1b52a9349d40bbcbf5e58f64f4ca21b94d2260bd7d WatchSource:0}: Error finding container 585628331ebab008c7bbdb1b52a9349d40bbcbf5e58f64f4ca21b94d2260bd7d: Status 404 returned error can't find the container with id 585628331ebab008c7bbdb1b52a9349d40bbcbf5e58f64f4ca21b94d2260bd7d
Oct 30 20:00:01.544302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.542986    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"]
Oct 30 20:00:01.575569 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:01.575530    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf3d12_c146_4ab3_b2b7_3c2fc1af5574.slice/crio-3c212e562ca82b72c8cc68fe098813fc80bb6e11be6d294b56e8281ff3d52b04 WatchSource:0}: Error finding container 3c212e562ca82b72c8cc68fe098813fc80bb6e11be6d294b56e8281ff3d52b04: Status 404 returned error can't find the container with id 3c212e562ca82b72c8cc68fe098813fc80bb6e11be6d294b56e8281ff3d52b04
Oct 30 20:00:01.584375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.583201    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"]
Oct 30 20:00:01.689162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.689082    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"]
Oct 30 20:00:01.743502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.743419    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" event={"ID":"badf3d12-c146-4ab3-b2b7-3c2fc1af5574","Type":"ContainerStarted","Data":"3c212e562ca82b72c8cc68fe098813fc80bb6e11be6d294b56e8281ff3d52b04"}
Oct 30 20:00:01.745326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.745294    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerStarted","Data":"4e25365d502ec47fdd9cee3c66e2c40e3589a24f38a5b97adea30d048ac10700"}
Oct 30 20:00:01.756399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.755814    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" event={"ID":"d7d4bc1d-85a2-4553-942d-dbc161fde6b7","Type":"ContainerStarted","Data":"1ade0933ab6653cad0977fe5cbc137535c95abd192efe1a8218dab7760910d86"}
Oct 30 20:00:01.761525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.760930    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" event={"ID":"2d2324b1-81c5-46d8-adec-73e2aebbe111","Type":"ContainerStarted","Data":"b41ef04ec7bd591643f46150849ed88055adb4a101669176677d8a9e1a400d48"}
Oct 30 20:00:01.764443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.764400    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" event={"ID":"07ec8783-2c39-4c35-836a-89a002783adb","Type":"ContainerStarted","Data":"230981213c60866d585b628bd83bd40d7cc2b65f9c1994f6633157879162c551"}
Oct 30 20:00:01.766717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.766685    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" event={"ID":"61dc526f-7dee-49ac-b389-b340ceab3266","Type":"ContainerStarted","Data":"585628331ebab008c7bbdb1b52a9349d40bbcbf5e58f64f4ca21b94d2260bd7d"}
Oct 30 20:00:01.849458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.849399    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"]
Oct 30 20:00:01.854296 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:01.854263    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cb5bde_e08d_496d_a83b_365d19d8d4a4.slice/crio-942e8b247081ae01233e0c2b31cbf5683826742ddb6a36df48ec5022beecb069 WatchSource:0}: Error finding container 942e8b247081ae01233e0c2b31cbf5683826742ddb6a36df48ec5022beecb069: Status 404 returned error can't find the container with id 942e8b247081ae01233e0c2b31cbf5683826742ddb6a36df48ec5022beecb069
Oct 30 20:00:01.911834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.911804    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"]
Oct 30 20:00:01.918036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.917996    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"]
Oct 30 20:00:01.976488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:01.976370    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"]
Oct 30 20:00:02.021680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.021641    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"]
Oct 30 20:00:02.067968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.067876    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"]
Oct 30 20:00:02.095781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.095688    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"]
Oct 30 20:00:02.102923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.102244    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"]
Oct 30 20:00:02.108066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.108027    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"]
Oct 30 20:00:02.166810 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:02.163728    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906f4700_7e26_4304_be69_bd6c77566780.slice/crio-119a70806baba014a7479d4a21da8fafb2bede2961af2b5abda1b80d79ed90b5 WatchSource:0}: Error finding container 119a70806baba014a7479d4a21da8fafb2bede2961af2b5abda1b80d79ed90b5: Status 404 returned error can't find the container with id 119a70806baba014a7479d4a21da8fafb2bede2961af2b5abda1b80d79ed90b5
Oct 30 20:00:02.172702 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:02.172665    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d4d50d_febd_43fe_90dc_cb8836687123.slice/crio-80b46487f6aea43c8f51e0265987167ff7a13590ccd8c54f922ff5f4409d84ea WatchSource:0}: Error finding container 80b46487f6aea43c8f51e0265987167ff7a13590ccd8c54f922ff5f4409d84ea: Status 404 returned error can't find the container with id 80b46487f6aea43c8f51e0265987167ff7a13590ccd8c54f922ff5f4409d84ea
Oct 30 20:00:02.182805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.182768    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"]
Oct 30 20:00:02.192676 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:02.191601    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32248e1e_4b52_4f73_b987_b0e348fab854.slice/crio-fdbd0b76f81503eec0be171def539a04e0da5a370ccf1e3f924c53f1b3b61a76 WatchSource:0}: Error finding container fdbd0b76f81503eec0be171def539a04e0da5a370ccf1e3f924c53f1b3b61a76: Status 404 returned error can't find the container with id fdbd0b76f81503eec0be171def539a04e0da5a370ccf1e3f924c53f1b3b61a76
Oct 30 20:00:02.195130 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:02.194709    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6456596_13e7_48e2_af8b_1ec739ca9871.slice/crio-b00e154c9713ce22d19f40fcd300757cde3283b67432e6de1b4930124013efdd WatchSource:0}: Error finding container b00e154c9713ce22d19f40fcd300757cde3283b67432e6de1b4930124013efdd: Status 404 returned error can't find the container with id b00e154c9713ce22d19f40fcd300757cde3283b67432e6de1b4930124013efdd
Oct 30 20:00:02.197995 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:02.197955    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60757fe_86c3_4d66_a25c_e1ce2a276453.slice/crio-dee500290c8b1a1e35854ee789bd19d12449d03b38c10e73d1c3eb8ee8b49816 WatchSource:0}: Error finding container dee500290c8b1a1e35854ee789bd19d12449d03b38c10e73d1c3eb8ee8b49816: Status 404 returned error can't find the container with id dee500290c8b1a1e35854ee789bd19d12449d03b38c10e73d1c3eb8ee8b49816
Oct 30 20:00:02.198435 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:02.198414    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032754c4_d986_4767_bc09_19ff1ba923a7.slice/crio-d430c3e956790c93ca707109252174f0745624f6da04bc45a475d9038f7bf880 WatchSource:0}: Error finding container d430c3e956790c93ca707109252174f0745624f6da04bc45a475d9038f7bf880: Status 404 returned error can't find the container with id d430c3e956790c93ca707109252174f0745624f6da04bc45a475d9038f7bf880
Oct 30 20:00:02.257329 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:02.257276    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:00:02.788076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.787988    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" event={"ID":"badf3d12-c146-4ab3-b2b7-3c2fc1af5574","Type":"ContainerStarted","Data":"820fdd122105da492c7ab7be1b5b50896f97421ca7798f3c116051d0e07cc1de"}
Oct 30 20:00:02.792318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.792193    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" event={"ID":"032754c4-d986-4767-bc09-19ff1ba923a7","Type":"ContainerStarted","Data":"d430c3e956790c93ca707109252174f0745624f6da04bc45a475d9038f7bf880"}
Oct 30 20:00:02.794379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.794344    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" event={"ID":"e6456596-13e7-48e2-af8b-1ec739ca9871","Type":"ContainerStarted","Data":"b00e154c9713ce22d19f40fcd300757cde3283b67432e6de1b4930124013efdd"}
Oct 30 20:00:02.797359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.796457    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerStarted","Data":"fe4f120c2d2e29a6871d5f10c51925787457037a0221d68ffc80fab4510d162c"}
Oct 30 20:00:02.798334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.798307    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerStarted","Data":"7d107e54509d80bf593dba22a83c8c74ee1dda471b78cc6ab3425637ef4bc4cb"}
Oct 30 20:00:02.800111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.800080    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerStarted","Data":"73a9569e5e2f8404f5866eb584d3d0635844e6f5c4d970f302fd68a146afa7d6"}
Oct 30 20:00:02.800232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.800135    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerStarted","Data":"fce41aad16dfeafb0fefcfef9bc20df71d68d8897bb4c80afb8d953949ba6c2d"}
Oct 30 20:00:02.802439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.802406    4519 generic.go:356] "Generic (PLEG): container finished" podID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerID="26613d3a79359fc79205bd9faabe768d6ab5a8835eb0acd97ea9d76ad16e7bf7" exitCode=60
Oct 30 20:00:02.802616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.802453    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" event={"ID":"d7d4bc1d-85a2-4553-942d-dbc161fde6b7","Type":"ContainerDied","Data":"26613d3a79359fc79205bd9faabe768d6ab5a8835eb0acd97ea9d76ad16e7bf7"}
Oct 30 20:00:02.802899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.802880    4519 scope.go:122] "RemoveContainer" containerID="26613d3a79359fc79205bd9faabe768d6ab5a8835eb0acd97ea9d76ad16e7bf7"
Oct 30 20:00:02.805136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.805106    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" event={"ID":"2d2324b1-81c5-46d8-adec-73e2aebbe111","Type":"ContainerStarted","Data":"62e7c526d59063d4e88c4a1e6368fd69566b8d84de1c1495d34ab372f8a2c388"}
Oct 30 20:00:02.807220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.807196    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" event={"ID":"fef72380-91f7-4631-8100-4454165123fc","Type":"ContainerStarted","Data":"ca96375b8630b42db5231c1ca661105b1473f106530bd8151e846daa22cbe360"}
Oct 30 20:00:02.809174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.809139    4519 generic.go:356] "Generic (PLEG): container finished" podID="07ec8783-2c39-4c35-836a-89a002783adb" containerID="00f51dce0eeed8b4d7df3eb322ba8b79e121c8d14dc21616293b41c75cfb8faa" exitCode=60
Oct 30 20:00:02.809297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.809211    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" event={"ID":"07ec8783-2c39-4c35-836a-89a002783adb","Type":"ContainerDied","Data":"00f51dce0eeed8b4d7df3eb322ba8b79e121c8d14dc21616293b41c75cfb8faa"}
Oct 30 20:00:02.809536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.809516    4519 scope.go:122] "RemoveContainer" containerID="00f51dce0eeed8b4d7df3eb322ba8b79e121c8d14dc21616293b41c75cfb8faa"
Oct 30 20:00:02.811202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.811175    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerStarted","Data":"70b6029d4ae3a36443be0f3db19eb49572a1a5638bb5a32ed6dd45c1d13200d4"}
Oct 30 20:00:02.811314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.811229    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerStarted","Data":"80b46487f6aea43c8f51e0265987167ff7a13590ccd8c54f922ff5f4409d84ea"}
Oct 30 20:00:02.812838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.812811    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerStarted","Data":"119a70806baba014a7479d4a21da8fafb2bede2961af2b5abda1b80d79ed90b5"}
Oct 30 20:00:02.815024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.814992    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" event={"ID":"61dc526f-7dee-49ac-b389-b340ceab3266","Type":"ContainerStarted","Data":"49b739bc66de6e788c04a187ecd65a16dde20c76dd4bfb57a933d3feffd5cd9d"}
Oct 30 20:00:02.819932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.818986    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" event={"ID":"32248e1e-4b52-4f73-b987-b0e348fab854","Type":"ContainerStarted","Data":"fdbd0b76f81503eec0be171def539a04e0da5a370ccf1e3f924c53f1b3b61a76"}
Oct 30 20:00:02.829571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.829438    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerStarted","Data":"2909e20203f544edd4b0dc512e78d63b2d5dc1d8d234301149aae6bf519b9374"}
Oct 30 20:00:02.829743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.829578    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerStarted","Data":"942e8b247081ae01233e0c2b31cbf5683826742ddb6a36df48ec5022beecb069"}
Oct 30 20:00:02.832845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.832799    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" event={"ID":"c60757fe-86c3-4d66-a25c-e1ce2a276453","Type":"ContainerStarted","Data":"dee500290c8b1a1e35854ee789bd19d12449d03b38c10e73d1c3eb8ee8b49816"}
Oct 30 20:00:02.835388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.835331    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" podStartSLOduration=2.835316299 podStartE2EDuration="2.835316299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.830204699 +0000 UTC m=+1527.330908424" watchObservedRunningTime="2026-10-30 20:00:02.835316299 +0000 UTC m=+1527.336020022"
Oct 30 20:00:02.861760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.861714    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:02.861917 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.861791    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:02.878893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.878814    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" podStartSLOduration=2.878792379 podStartE2EDuration="2.878792379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.874865203 +0000 UTC m=+1527.375568942" watchObservedRunningTime="2026-10-30 20:00:02.878792379 +0000 UTC m=+1527.379496096"
Oct 30 20:00:02.960981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:02.960873    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" podStartSLOduration=2.960854507 podStartE2EDuration="2.960854507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.959308601 +0000 UTC m=+1527.460012326" watchObservedRunningTime="2026-10-30 20:00:02.960854507 +0000 UTC m=+1527.461558232"
Oct 30 20:00:03.046204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.046142    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" podStartSLOduration=3.046123077 podStartE2EDuration="3.046123077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.043403554 +0000 UTC m=+1527.544107279" watchObservedRunningTime="2026-10-30 20:00:03.046123077 +0000 UTC m=+1527.546826800"
Oct 30 20:00:03.085578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.083752    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" podStartSLOduration=3.083731798 podStartE2EDuration="3.083731798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.082855013 +0000 UTC m=+1527.583558736" watchObservedRunningTime="2026-10-30 20:00:03.083731798 +0000 UTC m=+1527.584435523"
Oct 30 20:00:03.131188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.131138    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" podStartSLOduration=3.131121583 podStartE2EDuration="3.131121583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.12284912 +0000 UTC m=+1527.623552857" watchObservedRunningTime="2026-10-30 20:00:03.131121583 +0000 UTC m=+1527.631825306"
Oct 30 20:00:03.174509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.174438    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" podStartSLOduration=3.174420775 podStartE2EDuration="3.174420775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.170004073 +0000 UTC m=+1527.670707797" watchObservedRunningTime="2026-10-30 20:00:03.174420775 +0000 UTC m=+1527.675124499"
Oct 30 20:00:03.212855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.212798    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" podStartSLOduration=3.212782241 podStartE2EDuration="3.212782241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.210089898 +0000 UTC m=+1527.710793621" watchObservedRunningTime="2026-10-30 20:00:03.212782241 +0000 UTC m=+1527.713485964"
Oct 30 20:00:03.276760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.276677    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" podStartSLOduration=3.2766552190000002 podStartE2EDuration="3.276655219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.249079682 +0000 UTC m=+1527.749783411" watchObservedRunningTime="2026-10-30 20:00:03.276655219 +0000 UTC m=+1527.777358944"
Oct 30 20:00:03.312654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.312573    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" podStartSLOduration=3.31254221 podStartE2EDuration="3.31254221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.297184429 +0000 UTC m=+1527.797888154" watchObservedRunningTime="2026-10-30 20:00:03.31254221 +0000 UTC m=+1527.813245934"
Oct 30 20:00:03.843731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.843627    4519 generic.go:356] "Generic (PLEG): container finished" podID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerID="73a9569e5e2f8404f5866eb584d3d0635844e6f5c4d970f302fd68a146afa7d6" exitCode=60
Oct 30 20:00:03.844201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.843721    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerDied","Data":"73a9569e5e2f8404f5866eb584d3d0635844e6f5c4d970f302fd68a146afa7d6"}
Oct 30 20:00:03.844201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.844071    4519 scope.go:122] "RemoveContainer" containerID="73a9569e5e2f8404f5866eb584d3d0635844e6f5c4d970f302fd68a146afa7d6"
Oct 30 20:00:03.845879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.845855    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" event={"ID":"d7d4bc1d-85a2-4553-942d-dbc161fde6b7","Type":"ContainerStarted","Data":"5c353bd0cbbf14fec98e4f8e9b8a281950f406b4d072c4d0e10cdc7c4539c0e7"}
Oct 30 20:00:03.852116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.851082    4519 generic.go:356] "Generic (PLEG): container finished" podID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerID="62e7c526d59063d4e88c4a1e6368fd69566b8d84de1c1495d34ab372f8a2c388" exitCode=60
Oct 30 20:00:03.852116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.851610    4519 scope.go:122] "RemoveContainer" containerID="62e7c526d59063d4e88c4a1e6368fd69566b8d84de1c1495d34ab372f8a2c388"
Oct 30 20:00:03.852116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.851861    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" event={"ID":"2d2324b1-81c5-46d8-adec-73e2aebbe111","Type":"ContainerDied","Data":"62e7c526d59063d4e88c4a1e6368fd69566b8d84de1c1495d34ab372f8a2c388"}
Oct 30 20:00:03.857568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.853950    4519 generic.go:356] "Generic (PLEG): container finished" podID="fef72380-91f7-4631-8100-4454165123fc" containerID="f6342685b1158f9616afa11591c538b3ed4d35be2c3a0958fd292f0883f757c6" exitCode=60
Oct 30 20:00:03.857568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.854030    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" event={"ID":"fef72380-91f7-4631-8100-4454165123fc","Type":"ContainerDied","Data":"f6342685b1158f9616afa11591c538b3ed4d35be2c3a0958fd292f0883f757c6"}
Oct 30 20:00:03.857568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.854532    4519 scope.go:122] "RemoveContainer" containerID="f6342685b1158f9616afa11591c538b3ed4d35be2c3a0958fd292f0883f757c6"
Oct 30 20:00:03.858568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.858402    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" event={"ID":"07ec8783-2c39-4c35-836a-89a002783adb","Type":"ContainerStarted","Data":"47b62a84e9de3b829ae87d66ce873b17eace5ea957a2120b41358b0b73712a73"}
Oct 30 20:00:03.860663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.860638    4519 generic.go:356] "Generic (PLEG): container finished" podID="64d4d50d-febd-43fe-90dc-cb8836687123" containerID="70b6029d4ae3a36443be0f3db19eb49572a1a5638bb5a32ed6dd45c1d13200d4" exitCode=60
Oct 30 20:00:03.860793 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.860751    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerDied","Data":"70b6029d4ae3a36443be0f3db19eb49572a1a5638bb5a32ed6dd45c1d13200d4"}
Oct 30 20:00:03.861136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.861115    4519 scope.go:122] "RemoveContainer" containerID="70b6029d4ae3a36443be0f3db19eb49572a1a5638bb5a32ed6dd45c1d13200d4"
Oct 30 20:00:03.866932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.866908    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerStarted","Data":"6fd3ad34b24c996596d75bd4680b27fe6dd99dbcff8288ad1e0355dd458fdbfb"}
Oct 30 20:00:03.875103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.874925    4519 generic.go:356] "Generic (PLEG): container finished" podID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerID="a5212b395dd59dc1607dac435235de9815e5129ecedcaa7e9c202e660c3ce5d3" exitCode=0
Oct 30 20:00:03.875103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.874985    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" event={"ID":"d79bd922-ebb1-48e9-a300-9e0efb1b69d3","Type":"ContainerDied","Data":"a5212b395dd59dc1607dac435235de9815e5129ecedcaa7e9c202e660c3ce5d3"}
Oct 30 20:00:03.875671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.875643    4519 scope.go:122] "RemoveContainer" containerID="a5212b395dd59dc1607dac435235de9815e5129ecedcaa7e9c202e660c3ce5d3"
Oct 30 20:00:03.878668 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.878637    4519 generic.go:356] "Generic (PLEG): container finished" podID="61dc526f-7dee-49ac-b389-b340ceab3266" containerID="49b739bc66de6e788c04a187ecd65a16dde20c76dd4bfb57a933d3feffd5cd9d" exitCode=60
Oct 30 20:00:03.878799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.878745    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" event={"ID":"61dc526f-7dee-49ac-b389-b340ceab3266","Type":"ContainerDied","Data":"49b739bc66de6e788c04a187ecd65a16dde20c76dd4bfb57a933d3feffd5cd9d"}
Oct 30 20:00:03.879144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.879122    4519 scope.go:122] "RemoveContainer" containerID="49b739bc66de6e788c04a187ecd65a16dde20c76dd4bfb57a933d3feffd5cd9d"
Oct 30 20:00:03.884654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.881315    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:03.884654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.881461    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:03.884654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.881536    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:03.884654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.883796    4519 generic.go:356] "Generic (PLEG): container finished" podID="32248e1e-4b52-4f73-b987-b0e348fab854" containerID="3650758dd166956f0faef300877373e5883f2376cc999f4a1679c2cbbf6f6799" exitCode=60
Oct 30 20:00:03.884654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.883893    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" event={"ID":"32248e1e-4b52-4f73-b987-b0e348fab854","Type":"ContainerDied","Data":"3650758dd166956f0faef300877373e5883f2376cc999f4a1679c2cbbf6f6799"}
Oct 30 20:00:03.884654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.884451    4519 scope.go:122] "RemoveContainer" containerID="3650758dd166956f0faef300877373e5883f2376cc999f4a1679c2cbbf6f6799"
Oct 30 20:00:03.901749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.901703    4519 generic.go:356] "Generic (PLEG): container finished" podID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerID="2909e20203f544edd4b0dc512e78d63b2d5dc1d8d234301149aae6bf519b9374" exitCode=60
Oct 30 20:00:03.901980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.901862    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerDied","Data":"2909e20203f544edd4b0dc512e78d63b2d5dc1d8d234301149aae6bf519b9374"}
Oct 30 20:00:03.902327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.902308    4519 scope.go:122] "RemoveContainer" containerID="2909e20203f544edd4b0dc512e78d63b2d5dc1d8d234301149aae6bf519b9374"
Oct 30 20:00:03.905319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.905286    4519 generic.go:356] "Generic (PLEG): container finished" podID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerID="6f9113fc05e531eb1163aca6fb1c1e3b78d03eac5317320d26bfc1b84371431c" exitCode=60
Oct 30 20:00:03.905551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.905364    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" event={"ID":"c60757fe-86c3-4d66-a25c-e1ce2a276453","Type":"ContainerDied","Data":"6f9113fc05e531eb1163aca6fb1c1e3b78d03eac5317320d26bfc1b84371431c"}
Oct 30 20:00:03.905869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.905850    4519 scope.go:122] "RemoveContainer" containerID="6f9113fc05e531eb1163aca6fb1c1e3b78d03eac5317320d26bfc1b84371431c"
Oct 30 20:00:03.908560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.908532    4519 generic.go:356] "Generic (PLEG): container finished" podID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerID="820fdd122105da492c7ab7be1b5b50896f97421ca7798f3c116051d0e07cc1de" exitCode=60
Oct 30 20:00:03.908682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.908648    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" event={"ID":"badf3d12-c146-4ab3-b2b7-3c2fc1af5574","Type":"ContainerDied","Data":"820fdd122105da492c7ab7be1b5b50896f97421ca7798f3c116051d0e07cc1de"}
Oct 30 20:00:03.909012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.908942    4519 scope.go:122] "RemoveContainer" containerID="820fdd122105da492c7ab7be1b5b50896f97421ca7798f3c116051d0e07cc1de"
Oct 30 20:00:03.910696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.910672    4519 generic.go:356] "Generic (PLEG): container finished" podID="032754c4-d986-4767-bc09-19ff1ba923a7" containerID="e47037470b3429b2d08e63e662dbeac323f5e6103e714d30ddc12fcc2e5002d7" exitCode=60
Oct 30 20:00:03.910803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.910746    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" event={"ID":"032754c4-d986-4767-bc09-19ff1ba923a7","Type":"ContainerDied","Data":"e47037470b3429b2d08e63e662dbeac323f5e6103e714d30ddc12fcc2e5002d7"}
Oct 30 20:00:03.911098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.911076    4519 scope.go:122] "RemoveContainer" containerID="e47037470b3429b2d08e63e662dbeac323f5e6103e714d30ddc12fcc2e5002d7"
Oct 30 20:00:03.913186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.913144    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" event={"ID":"e6456596-13e7-48e2-af8b-1ec739ca9871","Type":"ContainerStarted","Data":"734ef00c2565e6eed8cb5f27ae55a0cea28e05fe76b3f9c1c3e224a1b9af0f0f"}
Oct 30 20:00:03.916309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.916197    4519 generic.go:356] "Generic (PLEG): container finished" podID="a278a416-54cd-4256-ac17-6e23b31c0671" containerID="fe4f120c2d2e29a6871d5f10c51925787457037a0221d68ffc80fab4510d162c" exitCode=60
Oct 30 20:00:03.916309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.916240    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerDied","Data":"fe4f120c2d2e29a6871d5f10c51925787457037a0221d68ffc80fab4510d162c"}
Oct 30 20:00:03.916699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.916671    4519 scope.go:122] "RemoveContainer" containerID="fe4f120c2d2e29a6871d5f10c51925787457037a0221d68ffc80fab4510d162c"
Oct 30 20:00:03.919259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.919207    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerStarted","Data":"ae86b9d8740fc5b3f8379cc02b404b7c7fc82ea3d260e7133287f331738b83fc"}
Oct 30 20:00:03.919729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.919708    4519 scope.go:122] "RemoveContainer" containerID="ae86b9d8740fc5b3f8379cc02b404b7c7fc82ea3d260e7133287f331738b83fc"
Oct 30 20:00:03.965145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:03.965085    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" podStartSLOduration=3.965064994 podStartE2EDuration="3.965064994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.962209103 +0000 UTC m=+1528.462912829" watchObservedRunningTime="2026-10-30 20:00:03.965064994 +0000 UTC m=+1528.465768719"
Oct 30 20:00:04.004174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:04.004115    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" podStartSLOduration=4.00409912 podStartE2EDuration="4.00409912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.999677818 +0000 UTC m=+1528.500381542" watchObservedRunningTime="2026-10-30 20:00:04.00409912 +0000 UTC m=+1528.504802844"
Oct 30 20:00:04.106069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:04.104593    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" podStartSLOduration=4.104577434 podStartE2EDuration="4.104577434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.10181943 +0000 UTC m=+1528.602523173" watchObservedRunningTime="2026-10-30 20:00:04.104577434 +0000 UTC m=+1528.605281157"
Oct 30 20:00:04.500162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:04.500091    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" podStartSLOduration=4.500074418 podStartE2EDuration="4.500074418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.498247353 +0000 UTC m=+1528.998951077" watchObservedRunningTime="2026-10-30 20:00:04.500074418 +0000 UTC m=+1529.000778138"
Oct 30 20:00:05.003677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.003638    4519 generic.go:356] "Generic (PLEG): container finished" podID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerID="5c353bd0cbbf14fec98e4f8e9b8a281950f406b4d072c4d0e10cdc7c4539c0e7" exitCode=60
Oct 30 20:00:05.004293 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.004267    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" event={"ID":"d7d4bc1d-85a2-4553-942d-dbc161fde6b7","Type":"ContainerDied","Data":"5c353bd0cbbf14fec98e4f8e9b8a281950f406b4d072c4d0e10cdc7c4539c0e7"}
Oct 30 20:00:05.004506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.004489    4519 scope.go:122] "RemoveContainer" containerID="26613d3a79359fc79205bd9faabe768d6ab5a8835eb0acd97ea9d76ad16e7bf7"
Oct 30 20:00:05.005168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.005149    4519 scope.go:122] "RemoveContainer" containerID="5c353bd0cbbf14fec98e4f8e9b8a281950f406b4d072c4d0e10cdc7c4539c0e7"
Oct 30 20:00:05.006207 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:05.006142    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx_odh-ai-gateway-infra(d7d4bc1d-85a2-4553-942d-dbc161fde6b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7"
Oct 30 20:00:05.013385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.010754    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" event={"ID":"2d2324b1-81c5-46d8-adec-73e2aebbe111","Type":"ContainerStarted","Data":"a512be4a0fadc897ceb0bb9efd61a243bf8d31a059f5e60490e6730c38b88f67"}
Oct 30 20:00:05.025266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.025226    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" event={"ID":"fef72380-91f7-4631-8100-4454165123fc","Type":"ContainerStarted","Data":"cb922a9603074845b3017093cb16a53b15992610bdcb7ae665335e39278490a7"}
Oct 30 20:00:05.032988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.031584    4519 generic.go:356] "Generic (PLEG): container finished" podID="07ec8783-2c39-4c35-836a-89a002783adb" containerID="47b62a84e9de3b829ae87d66ce873b17eace5ea957a2120b41358b0b73712a73" exitCode=60
Oct 30 20:00:05.032988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.031739    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" event={"ID":"07ec8783-2c39-4c35-836a-89a002783adb","Type":"ContainerDied","Data":"47b62a84e9de3b829ae87d66ce873b17eace5ea957a2120b41358b0b73712a73"}
Oct 30 20:00:05.032988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.032344    4519 scope.go:122] "RemoveContainer" containerID="47b62a84e9de3b829ae87d66ce873b17eace5ea957a2120b41358b0b73712a73"
Oct 30 20:00:05.032988 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:05.032894    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j_odh-ai-gateway-infra(07ec8783-2c39-4c35-836a-89a002783adb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" podUID="07ec8783-2c39-4c35-836a-89a002783adb"
Oct 30 20:00:05.038200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.036793    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerStarted","Data":"6cbc3690df74af6736f46bcd404467ccd5369b6c8451a052e285613ad4c2c52d"}
Oct 30 20:00:05.041876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.040906    4519 generic.go:356] "Generic (PLEG): container finished" podID="906f4700-7e26-4304-be69-bd6c77566780" containerID="6fd3ad34b24c996596d75bd4680b27fe6dd99dbcff8288ad1e0355dd458fdbfb" exitCode=60
Oct 30 20:00:05.041876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.041074    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerDied","Data":"6fd3ad34b24c996596d75bd4680b27fe6dd99dbcff8288ad1e0355dd458fdbfb"}
Oct 30 20:00:05.041876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.041661    4519 scope.go:122] "RemoveContainer" containerID="6fd3ad34b24c996596d75bd4680b27fe6dd99dbcff8288ad1e0355dd458fdbfb"
Oct 30 20:00:05.074256 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.071558    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"]
Oct 30 20:00:05.074256 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.073059    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" event={"ID":"d79bd922-ebb1-48e9-a300-9e0efb1b69d3","Type":"ContainerStarted","Data":"06c2f09d4ba76cd63af3fd4fffd477d41555448bf24658218a5b472230ebc62f"}
Oct 30 20:00:05.106119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.106081    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" event={"ID":"61dc526f-7dee-49ac-b389-b340ceab3266","Type":"ContainerStarted","Data":"1d2f65d306b60f5377ec28d04b7a1ff83705e1eaeb6ef11f641f87a7ed15c611"}
Oct 30 20:00:05.116592 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.116554    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8de508-6xksx"]
Oct 30 20:00:05.121065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.121005    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" event={"ID":"32248e1e-4b52-4f73-b987-b0e348fab854","Type":"ContainerStarted","Data":"ed12adbaac3267048c9034a35d2ac9acaad985633d6f891311fd4c6b69074227"}
Oct 30 20:00:05.127419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.127397    4519 scope.go:122] "RemoveContainer" containerID="00f51dce0eeed8b4d7df3eb322ba8b79e121c8d14dc21616293b41c75cfb8faa"
Oct 30 20:00:05.140269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.140172    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerStarted","Data":"9c72c9362c93d64da03a6f7827b26c9a6ca22d1a9640812ca9ca612ac390bfa2"}
Oct 30 20:00:05.169399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.169357    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" event={"ID":"c60757fe-86c3-4d66-a25c-e1ce2a276453","Type":"ContainerStarted","Data":"77c185db39abe14168ff210b1023b71adf73b9d9a52b20f090f3915c26b6a368"}
Oct 30 20:00:05.175171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.175139    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" event={"ID":"badf3d12-c146-4ab3-b2b7-3c2fc1af5574","Type":"ContainerStarted","Data":"7d7c3002113dedbe35ac831c09266928ab5962a68b032d1c2f00b595cac41a01"}
Oct 30 20:00:05.184386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.184316    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" event={"ID":"032754c4-d986-4767-bc09-19ff1ba923a7","Type":"ContainerStarted","Data":"734a84e2eabfffce1a57c7981c7db7e428cca6e4bf10fc1dece9f6a1483ac7bd"}
Oct 30 20:00:05.194963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.194026    4519 generic.go:356] "Generic (PLEG): container finished" podID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerID="734ef00c2565e6eed8cb5f27ae55a0cea28e05fe76b3f9c1c3e224a1b9af0f0f" exitCode=60
Oct 30 20:00:05.195989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.195783    4519 scope.go:122] "RemoveContainer" containerID="734ef00c2565e6eed8cb5f27ae55a0cea28e05fe76b3f9c1c3e224a1b9af0f0f"
Oct 30 20:00:05.196875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.195808    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" event={"ID":"e6456596-13e7-48e2-af8b-1ec739ca9871","Type":"ContainerDied","Data":"734ef00c2565e6eed8cb5f27ae55a0cea28e05fe76b3f9c1c3e224a1b9af0f0f"}
Oct 30 20:00:05.203677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.203633    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerStarted","Data":"b3805b0bd337ef8599681a15e449978cf190828644cb338c2256c0a2b167e1d4"}
Oct 30 20:00:05.220678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.219757    4519 generic.go:356] "Generic (PLEG): container finished" podID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerID="ae86b9d8740fc5b3f8379cc02b404b7c7fc82ea3d260e7133287f331738b83fc" exitCode=60
Oct 30 20:00:05.220678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.219855    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerDied","Data":"ae86b9d8740fc5b3f8379cc02b404b7c7fc82ea3d260e7133287f331738b83fc"}
Oct 30 20:00:05.220678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.219892    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerStarted","Data":"b4c56e9b9c92d966626d80faa7342603fd8c5c944a4709a3226915be8a09aebc"}
Oct 30 20:00:05.222165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.222131    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerStarted","Data":"409b6990b0436f67a5e2f329aef167653c363a50267727dbd6dcacf177914f0a"}
Oct 30 20:00:05.611202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.611148    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" podStartSLOduration=5.611134043 podStartE2EDuration="5.611134043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:05.609592403 +0000 UTC m=+1530.110296127" watchObservedRunningTime="2026-10-30 20:00:05.611134043 +0000 UTC m=+1530.111837800"
Oct 30 20:00:05.616481 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.616434    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b" path="/var/lib/kubelet/pods/e24c6ec9-82a9-4e4c-bec7-630f35e0cb7b/volumes"
Oct 30 20:00:05.710053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.709975    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" podStartSLOduration=5.709957428 podStartE2EDuration="5.709957428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:05.704234609 +0000 UTC m=+1530.204938334" watchObservedRunningTime="2026-10-30 20:00:05.709957428 +0000 UTC m=+1530.210661186"
Oct 30 20:00:05.983914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.983879    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 20:00:05.984917 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.984881    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:00:05.985083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:05.984944    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 20:00:06.234043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.234005    4519 generic.go:356] "Generic (PLEG): container finished" podID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerID="77c185db39abe14168ff210b1023b71adf73b9d9a52b20f090f3915c26b6a368" exitCode=60
Oct 30 20:00:06.234586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.234085    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" event={"ID":"c60757fe-86c3-4d66-a25c-e1ce2a276453","Type":"ContainerDied","Data":"77c185db39abe14168ff210b1023b71adf73b9d9a52b20f090f3915c26b6a368"}
Oct 30 20:00:06.234586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.234147    4519 scope.go:122] "RemoveContainer" containerID="6f9113fc05e531eb1163aca6fb1c1e3b78d03eac5317320d26bfc1b84371431c"
Oct 30 20:00:06.234780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.234762    4519 scope.go:122] "RemoveContainer" containerID="77c185db39abe14168ff210b1023b71adf73b9d9a52b20f090f3915c26b6a368"
Oct 30 20:00:06.235297 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.235167    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb_odh-ai-gateway-infra(c60757fe-86c3-4d66-a25c-e1ce2a276453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453"
Oct 30 20:00:06.236479 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.236447    4519 generic.go:356] "Generic (PLEG): container finished" podID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerID="7d7c3002113dedbe35ac831c09266928ab5962a68b032d1c2f00b595cac41a01" exitCode=60
Oct 30 20:00:06.236562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.236499    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" event={"ID":"badf3d12-c146-4ab3-b2b7-3c2fc1af5574","Type":"ContainerDied","Data":"7d7c3002113dedbe35ac831c09266928ab5962a68b032d1c2f00b595cac41a01"}
Oct 30 20:00:06.236886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.236868    4519 scope.go:122] "RemoveContainer" containerID="7d7c3002113dedbe35ac831c09266928ab5962a68b032d1c2f00b595cac41a01"
Oct 30 20:00:06.237241 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.237221    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-bbbl2_odh-ai-gateway-infra(badf3d12-c146-4ab3-b2b7-3c2fc1af5574)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574"
Oct 30 20:00:06.238752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.238727    4519 generic.go:356] "Generic (PLEG): container finished" podID="032754c4-d986-4767-bc09-19ff1ba923a7" containerID="734a84e2eabfffce1a57c7981c7db7e428cca6e4bf10fc1dece9f6a1483ac7bd" exitCode=60
Oct 30 20:00:06.238858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.238835    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" event={"ID":"032754c4-d986-4767-bc09-19ff1ba923a7","Type":"ContainerDied","Data":"734a84e2eabfffce1a57c7981c7db7e428cca6e4bf10fc1dece9f6a1483ac7bd"}
Oct 30 20:00:06.239075 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.239059    4519 scope.go:122] "RemoveContainer" containerID="734a84e2eabfffce1a57c7981c7db7e428cca6e4bf10fc1dece9f6a1483ac7bd"
Oct 30 20:00:06.239389 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.239368    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d_odh-ai-gateway-infra(032754c4-d986-4767-bc09-19ff1ba923a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" podUID="032754c4-d986-4767-bc09-19ff1ba923a7"
Oct 30 20:00:06.241091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.241068    4519 generic.go:356] "Generic (PLEG): container finished" podID="a278a416-54cd-4256-ac17-6e23b31c0671" containerID="b3805b0bd337ef8599681a15e449978cf190828644cb338c2256c0a2b167e1d4" exitCode=60
Oct 30 20:00:06.241196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.241143    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerDied","Data":"b3805b0bd337ef8599681a15e449978cf190828644cb338c2256c0a2b167e1d4"}
Oct 30 20:00:06.241608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.241586    4519 scope.go:122] "RemoveContainer" containerID="b3805b0bd337ef8599681a15e449978cf190828644cb338c2256c0a2b167e1d4"
Oct 30 20:00:06.241961 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.241938    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5_odh-ai-gateway-infra(a278a416-54cd-4256-ac17-6e23b31c0671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" podUID="a278a416-54cd-4256-ac17-6e23b31c0671"
Oct 30 20:00:06.243523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.243459    4519 generic.go:356] "Generic (PLEG): container finished" podID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerID="b4c56e9b9c92d966626d80faa7342603fd8c5c944a4709a3226915be8a09aebc" exitCode=60
Oct 30 20:00:06.243623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.243527    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerDied","Data":"b4c56e9b9c92d966626d80faa7342603fd8c5c944a4709a3226915be8a09aebc"}
Oct 30 20:00:06.243867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.243848    4519 scope.go:122] "RemoveContainer" containerID="b4c56e9b9c92d966626d80faa7342603fd8c5c944a4709a3226915be8a09aebc"
Oct 30 20:00:06.244186 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.244166    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k_odh-ai-gateway-infra(2ea9d0ec-fb50-4c79-b2d0-1ec024506b59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59"
Oct 30 20:00:06.245854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.245833    4519 generic.go:356] "Generic (PLEG): container finished" podID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerID="409b6990b0436f67a5e2f329aef167653c363a50267727dbd6dcacf177914f0a" exitCode=60
Oct 30 20:00:06.245958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.245908    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerDied","Data":"409b6990b0436f67a5e2f329aef167653c363a50267727dbd6dcacf177914f0a"}
Oct 30 20:00:06.246157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.246139    4519 scope.go:122] "RemoveContainer" containerID="409b6990b0436f67a5e2f329aef167653c363a50267727dbd6dcacf177914f0a"
Oct 30 20:00:06.246462 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.246439    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t_odh-ai-gateway-infra(33b31c7a-ff39-4203-9e69-e8b773b009e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4"
Oct 30 20:00:06.290001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.289942    4519 scope.go:122] "RemoveContainer" containerID="5c353bd0cbbf14fec98e4f8e9b8a281950f406b4d072c4d0e10cdc7c4539c0e7"
Oct 30 20:00:06.290440 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.290409    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx_odh-ai-gateway-infra(d7d4bc1d-85a2-4553-942d-dbc161fde6b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7"
Oct 30 20:00:06.290868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.290561    4519 scope.go:122] "RemoveContainer" containerID="820fdd122105da492c7ab7be1b5b50896f97421ca7798f3c116051d0e07cc1de"
Oct 30 20:00:06.292568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.292535    4519 generic.go:356] "Generic (PLEG): container finished" podID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerID="a512be4a0fadc897ceb0bb9efd61a243bf8d31a059f5e60490e6730c38b88f67" exitCode=60
Oct 30 20:00:06.292662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.292622    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" event={"ID":"2d2324b1-81c5-46d8-adec-73e2aebbe111","Type":"ContainerDied","Data":"a512be4a0fadc897ceb0bb9efd61a243bf8d31a059f5e60490e6730c38b88f67"}
Oct 30 20:00:06.293029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.293004    4519 scope.go:122] "RemoveContainer" containerID="a512be4a0fadc897ceb0bb9efd61a243bf8d31a059f5e60490e6730c38b88f67"
Oct 30 20:00:06.293479 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.293444    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8_odh-ai-gateway-infra(2d2324b1-81c5-46d8-adec-73e2aebbe111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111"
Oct 30 20:00:06.294831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.294809    4519 generic.go:356] "Generic (PLEG): container finished" podID="fef72380-91f7-4631-8100-4454165123fc" containerID="cb922a9603074845b3017093cb16a53b15992610bdcb7ae665335e39278490a7" exitCode=60
Oct 30 20:00:06.294939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.294894    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" event={"ID":"fef72380-91f7-4631-8100-4454165123fc","Type":"ContainerDied","Data":"cb922a9603074845b3017093cb16a53b15992610bdcb7ae665335e39278490a7"}
Oct 30 20:00:06.295206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.295192    4519 scope.go:122] "RemoveContainer" containerID="cb922a9603074845b3017093cb16a53b15992610bdcb7ae665335e39278490a7"
Oct 30 20:00:06.295621 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.295596    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr_odh-ai-gateway-infra(fef72380-91f7-4631-8100-4454165123fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" podUID="fef72380-91f7-4631-8100-4454165123fc"
Oct 30 20:00:06.297863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.297841    4519 scope.go:122] "RemoveContainer" containerID="47b62a84e9de3b829ae87d66ce873b17eace5ea957a2120b41358b0b73712a73"
Oct 30 20:00:06.298265 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.298239    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j_odh-ai-gateway-infra(07ec8783-2c39-4c35-836a-89a002783adb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" podUID="07ec8783-2c39-4c35-836a-89a002783adb"
Oct 30 20:00:06.300239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.300211    4519 generic.go:356] "Generic (PLEG): container finished" podID="64d4d50d-febd-43fe-90dc-cb8836687123" containerID="6cbc3690df74af6736f46bcd404467ccd5369b6c8451a052e285613ad4c2c52d" exitCode=60
Oct 30 20:00:06.300356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.300269    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerDied","Data":"6cbc3690df74af6736f46bcd404467ccd5369b6c8451a052e285613ad4c2c52d"}
Oct 30 20:00:06.300702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.300680    4519 scope.go:122] "RemoveContainer" containerID="6cbc3690df74af6736f46bcd404467ccd5369b6c8451a052e285613ad4c2c52d"
Oct 30 20:00:06.301104 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.301076    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6_odh-ai-gateway-infra(64d4d50d-febd-43fe-90dc-cb8836687123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" podUID="64d4d50d-febd-43fe-90dc-cb8836687123"
Oct 30 20:00:06.304849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.304820    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerStarted","Data":"16851ef75bb578136fee2e66dc11b39c5cb262602d7627177193d9b3197dffef"}
Oct 30 20:00:06.307910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.307882    4519 generic.go:356] "Generic (PLEG): container finished" podID="61dc526f-7dee-49ac-b389-b340ceab3266" containerID="1d2f65d306b60f5377ec28d04b7a1ff83705e1eaeb6ef11f641f87a7ed15c611" exitCode=60
Oct 30 20:00:06.308052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.307967    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" event={"ID":"61dc526f-7dee-49ac-b389-b340ceab3266","Type":"ContainerDied","Data":"1d2f65d306b60f5377ec28d04b7a1ff83705e1eaeb6ef11f641f87a7ed15c611"}
Oct 30 20:00:06.308676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.308658    4519 scope.go:122] "RemoveContainer" containerID="1d2f65d306b60f5377ec28d04b7a1ff83705e1eaeb6ef11f641f87a7ed15c611"
Oct 30 20:00:06.309258 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.309231    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4_odh-ai-gateway-infra(61dc526f-7dee-49ac-b389-b340ceab3266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" podUID="61dc526f-7dee-49ac-b389-b340ceab3266"
Oct 30 20:00:06.310302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.310281    4519 scope.go:122] "RemoveContainer" containerID="e47037470b3429b2d08e63e662dbeac323f5e6103e714d30ddc12fcc2e5002d7"
Oct 30 20:00:06.310509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.310444    4519 generic.go:356] "Generic (PLEG): container finished" podID="32248e1e-4b52-4f73-b987-b0e348fab854" containerID="ed12adbaac3267048c9034a35d2ac9acaad985633d6f891311fd4c6b69074227" exitCode=60
Oct 30 20:00:06.310601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.310553    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" event={"ID":"32248e1e-4b52-4f73-b987-b0e348fab854","Type":"ContainerDied","Data":"ed12adbaac3267048c9034a35d2ac9acaad985633d6f891311fd4c6b69074227"}
Oct 30 20:00:06.310885 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.310864    4519 scope.go:122] "RemoveContainer" containerID="ed12adbaac3267048c9034a35d2ac9acaad985633d6f891311fd4c6b69074227"
Oct 30 20:00:06.311299 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.311274    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk_odh-ai-gateway-infra(32248e1e-4b52-4f73-b987-b0e348fab854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" podUID="32248e1e-4b52-4f73-b987-b0e348fab854"
Oct 30 20:00:06.313611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.313584    4519 generic.go:356] "Generic (PLEG): container finished" podID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerID="9c72c9362c93d64da03a6f7827b26c9a6ca22d1a9640812ca9ca612ac390bfa2" exitCode=60
Oct 30 20:00:06.313802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.313664    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerDied","Data":"9c72c9362c93d64da03a6f7827b26c9a6ca22d1a9640812ca9ca612ac390bfa2"}
Oct 30 20:00:06.314029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.314009    4519 scope.go:122] "RemoveContainer" containerID="9c72c9362c93d64da03a6f7827b26c9a6ca22d1a9640812ca9ca612ac390bfa2"
Oct 30 20:00:06.314405 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:06.314380    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc_odh-ai-gateway-infra(32cb5bde-e08d-496d-a83b-365d19d8d4a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4"
Oct 30 20:00:06.326572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.326549    4519 scope.go:122] "RemoveContainer" containerID="fe4f120c2d2e29a6871d5f10c51925787457037a0221d68ffc80fab4510d162c"
Oct 30 20:00:06.342198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.342172    4519 scope.go:122] "RemoveContainer" containerID="ae86b9d8740fc5b3f8379cc02b404b7c7fc82ea3d260e7133287f331738b83fc"
Oct 30 20:00:06.400940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.400915    4519 scope.go:122] "RemoveContainer" containerID="73a9569e5e2f8404f5866eb584d3d0635844e6f5c4d970f302fd68a146afa7d6"
Oct 30 20:00:06.417342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.417313    4519 scope.go:122] "RemoveContainer" containerID="62e7c526d59063d4e88c4a1e6368fd69566b8d84de1c1495d34ab372f8a2c388"
Oct 30 20:00:06.436283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.436215    4519 scope.go:122] "RemoveContainer" containerID="f6342685b1158f9616afa11591c538b3ed4d35be2c3a0958fd292f0883f757c6"
Oct 30 20:00:06.488778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.488751    4519 scope.go:122] "RemoveContainer" containerID="70b6029d4ae3a36443be0f3db19eb49572a1a5638bb5a32ed6dd45c1d13200d4"
Oct 30 20:00:06.506763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.506732    4519 scope.go:122] "RemoveContainer" containerID="49b739bc66de6e788c04a187ecd65a16dde20c76dd4bfb57a933d3feffd5cd9d"
Oct 30 20:00:06.522750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.522727    4519 scope.go:122] "RemoveContainer" containerID="3650758dd166956f0faef300877373e5883f2376cc999f4a1679c2cbbf6f6799"
Oct 30 20:00:06.539515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.539487    4519 scope.go:122] "RemoveContainer" containerID="2909e20203f544edd4b0dc512e78d63b2d5dc1d8d234301149aae6bf519b9374"
Oct 30 20:00:06.985564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.982965    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:00:06.985795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:06.985601    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 20:00:07.323801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.323767    4519 scope.go:122] "RemoveContainer" containerID="7d7c3002113dedbe35ac831c09266928ab5962a68b032d1c2f00b595cac41a01"
Oct 30 20:00:07.324249 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.324211    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-bbbl2_odh-ai-gateway-infra(badf3d12-c146-4ab3-b2b7-3c2fc1af5574)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574"
Oct 30 20:00:07.325545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.325520    4519 scope.go:122] "RemoveContainer" containerID="734a84e2eabfffce1a57c7981c7db7e428cca6e4bf10fc1dece9f6a1483ac7bd"
Oct 30 20:00:07.325939 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.325902    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d_odh-ai-gateway-infra(032754c4-d986-4767-bc09-19ff1ba923a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" podUID="032754c4-d986-4767-bc09-19ff1ba923a7"
Oct 30 20:00:07.327129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.327102    4519 generic.go:356] "Generic (PLEG): container finished" podID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerID="16520161fef153a49672fa9211a3d8975fcec90cd0d5c004f2645e93b1ca3bed" exitCode=60
Oct 30 20:00:07.327223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.327182    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" event={"ID":"e6456596-13e7-48e2-af8b-1ec739ca9871","Type":"ContainerDied","Data":"16520161fef153a49672fa9211a3d8975fcec90cd0d5c004f2645e93b1ca3bed"}
Oct 30 20:00:07.327302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.327255    4519 scope.go:122] "RemoveContainer" containerID="734ef00c2565e6eed8cb5f27ae55a0cea28e05fe76b3f9c1c3e224a1b9af0f0f"
Oct 30 20:00:07.327538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.327518    4519 scope.go:122] "RemoveContainer" containerID="16520161fef153a49672fa9211a3d8975fcec90cd0d5c004f2645e93b1ca3bed"
Oct 30 20:00:07.327939 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.327916    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv_odh-ai-gateway-infra(e6456596-13e7-48e2-af8b-1ec739ca9871)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871"
Oct 30 20:00:07.329188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.329160    4519 scope.go:122] "RemoveContainer" containerID="b3805b0bd337ef8599681a15e449978cf190828644cb338c2256c0a2b167e1d4"
Oct 30 20:00:07.329619 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.329597    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5_odh-ai-gateway-infra(a278a416-54cd-4256-ac17-6e23b31c0671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" podUID="a278a416-54cd-4256-ac17-6e23b31c0671"
Oct 30 20:00:07.330851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.330830    4519 scope.go:122] "RemoveContainer" containerID="b4c56e9b9c92d966626d80faa7342603fd8c5c944a4709a3226915be8a09aebc"
Oct 30 20:00:07.331128 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.331110    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k_odh-ai-gateway-infra(2ea9d0ec-fb50-4c79-b2d0-1ec024506b59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59"
Oct 30 20:00:07.332858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.332840    4519 scope.go:122] "RemoveContainer" containerID="409b6990b0436f67a5e2f329aef167653c363a50267727dbd6dcacf177914f0a"
Oct 30 20:00:07.333153 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.333133    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t_odh-ai-gateway-infra(33b31c7a-ff39-4203-9e69-e8b773b009e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4"
Oct 30 20:00:07.334677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.334657    4519 scope.go:122] "RemoveContainer" containerID="a512be4a0fadc897ceb0bb9efd61a243bf8d31a059f5e60490e6730c38b88f67"
Oct 30 20:00:07.334990 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.334968    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8_odh-ai-gateway-infra(2d2324b1-81c5-46d8-adec-73e2aebbe111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111"
Oct 30 20:00:07.336683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.336664    4519 scope.go:122] "RemoveContainer" containerID="cb922a9603074845b3017093cb16a53b15992610bdcb7ae665335e39278490a7"
Oct 30 20:00:07.336948 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.336930    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr_odh-ai-gateway-infra(fef72380-91f7-4631-8100-4454165123fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" podUID="fef72380-91f7-4631-8100-4454165123fc"
Oct 30 20:00:07.338610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.338587    4519 scope.go:122] "RemoveContainer" containerID="6cbc3690df74af6736f46bcd404467ccd5369b6c8451a052e285613ad4c2c52d"
Oct 30 20:00:07.338895 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.338875    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6_odh-ai-gateway-infra(64d4d50d-febd-43fe-90dc-cb8836687123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" podUID="64d4d50d-febd-43fe-90dc-cb8836687123"
Oct 30 20:00:07.340352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.340325    4519 generic.go:356] "Generic (PLEG): container finished" podID="906f4700-7e26-4304-be69-bd6c77566780" containerID="16851ef75bb578136fee2e66dc11b39c5cb262602d7627177193d9b3197dffef" exitCode=60
Oct 30 20:00:07.340426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.340403    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerDied","Data":"16851ef75bb578136fee2e66dc11b39c5cb262602d7627177193d9b3197dffef"}
Oct 30 20:00:07.340642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.340620    4519 scope.go:122] "RemoveContainer" containerID="16851ef75bb578136fee2e66dc11b39c5cb262602d7627177193d9b3197dffef"
Oct 30 20:00:07.340927 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.340907    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx_odh-ai-gateway-infra(906f4700-7e26-4304-be69-bd6c77566780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" podUID="906f4700-7e26-4304-be69-bd6c77566780"
Oct 30 20:00:07.342424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.342406    4519 scope.go:122] "RemoveContainer" containerID="1d2f65d306b60f5377ec28d04b7a1ff83705e1eaeb6ef11f641f87a7ed15c611"
Oct 30 20:00:07.342696 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.342676    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4_odh-ai-gateway-infra(61dc526f-7dee-49ac-b389-b340ceab3266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" podUID="61dc526f-7dee-49ac-b389-b340ceab3266"
Oct 30 20:00:07.343921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.343899    4519 scope.go:122] "RemoveContainer" containerID="6fd3ad34b24c996596d75bd4680b27fe6dd99dbcff8288ad1e0355dd458fdbfb"
Oct 30 20:00:07.344146 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.344127    4519 scope.go:122] "RemoveContainer" containerID="ed12adbaac3267048c9034a35d2ac9acaad985633d6f891311fd4c6b69074227"
Oct 30 20:00:07.344413 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.344377    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk_odh-ai-gateway-infra(32248e1e-4b52-4f73-b987-b0e348fab854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" podUID="32248e1e-4b52-4f73-b987-b0e348fab854"
Oct 30 20:00:07.345920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.345900    4519 scope.go:122] "RemoveContainer" containerID="9c72c9362c93d64da03a6f7827b26c9a6ca22d1a9640812ca9ca612ac390bfa2"
Oct 30 20:00:07.346150 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.346127    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc_odh-ai-gateway-infra(32cb5bde-e08d-496d-a83b-365d19d8d4a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4"
Oct 30 20:00:07.372493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.368206    4519 scope.go:122] "RemoveContainer" containerID="77c185db39abe14168ff210b1023b71adf73b9d9a52b20f090f3915c26b6a368"
Oct 30 20:00:07.372493 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:07.368696    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb_odh-ai-gateway-infra(c60757fe-86c3-4d66-a25c-e1ce2a276453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453"
Oct 30 20:00:07.981942 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.981901    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:00:07.982164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:07.981975    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 20:00:08.406009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:08.405914    4519 scope.go:122] "RemoveContainer" containerID="16520161fef153a49672fa9211a3d8975fcec90cd0d5c004f2645e93b1ca3bed"
Oct 30 20:00:08.406515 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:08.406430    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv_odh-ai-gateway-infra(e6456596-13e7-48e2-af8b-1ec739ca9871)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871"
Oct 30 20:00:08.408190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:08.408168    4519 scope.go:122] "RemoveContainer" containerID="16851ef75bb578136fee2e66dc11b39c5cb262602d7627177193d9b3197dffef"
Oct 30 20:00:08.408435 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:08.408419    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx_odh-ai-gateway-infra(906f4700-7e26-4304-be69-bd6c77566780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" podUID="906f4700-7e26-4304-be69-bd6c77566780"
Oct 30 20:00:08.981494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:08.981085    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:00:08.981494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:08.981136    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 20:00:09.982134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:09.982087    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:00:09.982573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:09.982158    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 20:00:10.999589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:10.999549    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 20:00:11.000531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:11.000503    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 20:00:11.003331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:11.003303    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 20:00:13.321594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.321557    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"]
Oct 30 20:00:13.328412 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.328376    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.343343 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.343309    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-652137-openshift-default-dockercfg-6plw8\""
Oct 30 20:00:13.385534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.383557    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"]
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.482716    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-envoy\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.482772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-data\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.482866    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-podinfo\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.482924    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgz6q\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-kube-api-access-wgz6q\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.482967    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-token\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.483047    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.483114    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-certs\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.483222    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istiod-ca-cert\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.494489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.483254    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-credential-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.586851    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-envoy\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.586909    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-data\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587021    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-podinfo\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587047    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgz6q\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-kube-api-access-wgz6q\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587091    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-token\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587202    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587270    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-certs\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587384    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istiod-ca-cert\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587420    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-credential-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.590680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.587941    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-credential-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.679558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.643355    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-data\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.679558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.647893    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-certs\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.679558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.648131    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-socket\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.679558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.652232    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-podinfo\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.679558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.653150    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-envoy\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.727485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.724128    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istiod-ca-cert\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.808001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.798424    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-token\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.934489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.925710    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgz6q\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-kube-api-access-wgz6q\") pod \"e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") " pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:13.956185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:13.955820    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:14.584926 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:14.584889    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247c9ec7_c64a_4e57_96a1_0a26a7cf1a37.slice/crio-8a5be9bd26f6a2af19fd94b64365f9add5d9038f3a85750c0706b3a020d16d41 WatchSource:0}: Error finding container 8a5be9bd26f6a2af19fd94b64365f9add5d9038f3a85750c0706b3a020d16d41: Status 404 returned error can't find the container with id 8a5be9bd26f6a2af19fd94b64365f9add5d9038f3a85750c0706b3a020d16d41
Oct 30 20:00:14.585304 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.585160    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"]
Oct 30 20:00:14.590018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.589982    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:14.590272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.590237    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:14.590371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.590308    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:14.782715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.776615    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" event={"ID":"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37","Type":"ContainerStarted","Data":"bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f"}
Oct 30 20:00:14.782715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.776696    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" event={"ID":"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37","Type":"ContainerStarted","Data":"8a5be9bd26f6a2af19fd94b64365f9add5d9038f3a85750c0706b3a020d16d41"}
Oct 30 20:00:14.854959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.851876    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podStartSLOduration=1.8518515629999999 podStartE2EDuration="1.851851563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:14.84383386 +0000 UTC m=+1539.344537597" watchObservedRunningTime="2026-10-30 20:00:14.851851563 +0000 UTC m=+1539.352555290"
Oct 30 20:00:14.959493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:14.958624    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:15.611665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:15.611628    4519 scope.go:122] "RemoveContainer" containerID="a512be4a0fadc897ceb0bb9efd61a243bf8d31a059f5e60490e6730c38b88f67"
Oct 30 20:00:15.612161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:15.611768    4519 scope.go:122] "RemoveContainer" containerID="47b62a84e9de3b829ae87d66ce873b17eace5ea957a2120b41358b0b73712a73"
Oct 30 20:00:15.612161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:15.611859    4519 scope.go:122] "RemoveContainer" containerID="1d2f65d306b60f5377ec28d04b7a1ff83705e1eaeb6ef11f641f87a7ed15c611"
Oct 30 20:00:15.963609 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:15.963570    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:16.607296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.607251    4519 scope.go:122] "RemoveContainer" containerID="cb922a9603074845b3017093cb16a53b15992610bdcb7ae665335e39278490a7"
Oct 30 20:00:16.630262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.630226    4519 scope.go:122] "RemoveContainer" containerID="409b6990b0436f67a5e2f329aef167653c363a50267727dbd6dcacf177914f0a"
Oct 30 20:00:16.797192 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.797159    4519 generic.go:356] "Generic (PLEG): container finished" podID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerID="b12dceb40924d24979a596c3f5ea84d146b748f405dc29d27220d82eb6bce59c" exitCode=60
Oct 30 20:00:16.797331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.797232    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" event={"ID":"2d2324b1-81c5-46d8-adec-73e2aebbe111","Type":"ContainerDied","Data":"b12dceb40924d24979a596c3f5ea84d146b748f405dc29d27220d82eb6bce59c"}
Oct 30 20:00:16.797331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.797296    4519 scope.go:122] "RemoveContainer" containerID="a512be4a0fadc897ceb0bb9efd61a243bf8d31a059f5e60490e6730c38b88f67"
Oct 30 20:00:16.797841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.797815    4519 scope.go:122] "RemoveContainer" containerID="b12dceb40924d24979a596c3f5ea84d146b748f405dc29d27220d82eb6bce59c"
Oct 30 20:00:16.798391 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:16.798290    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8_odh-ai-gateway-infra(2d2324b1-81c5-46d8-adec-73e2aebbe111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111"
Oct 30 20:00:16.799915 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.799893    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" event={"ID":"07ec8783-2c39-4c35-836a-89a002783adb","Type":"ContainerStarted","Data":"4995a474cf2525222493adbd4d5c352b3e520e00bbece77ac6028cda5d819889"}
Oct 30 20:00:16.802019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.801990    4519 generic.go:356] "Generic (PLEG): container finished" podID="61dc526f-7dee-49ac-b389-b340ceab3266" containerID="3e64ec9453bf1217038049bd8db1595f9e8e3e43e9415ff6804752d8e34255ab" exitCode=60
Oct 30 20:00:16.802118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.802058    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" event={"ID":"61dc526f-7dee-49ac-b389-b340ceab3266","Type":"ContainerDied","Data":"3e64ec9453bf1217038049bd8db1595f9e8e3e43e9415ff6804752d8e34255ab"}
Oct 30 20:00:16.802314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.802296    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:16.802657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.802636    4519 scope.go:122] "RemoveContainer" containerID="3e64ec9453bf1217038049bd8db1595f9e8e3e43e9415ff6804752d8e34255ab"
Oct 30 20:00:16.802941 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:16.802918    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4_odh-ai-gateway-infra(61dc526f-7dee-49ac-b389-b340ceab3266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" podUID="61dc526f-7dee-49ac-b389-b340ceab3266"
Oct 30 20:00:16.803276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.803253    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:00:16.826268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:16.826240    4519 scope.go:122] "RemoveContainer" containerID="1d2f65d306b60f5377ec28d04b7a1ff83705e1eaeb6ef11f641f87a7ed15c611"
Oct 30 20:00:17.606011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.605852    4519 scope.go:122] "RemoveContainer" containerID="ed12adbaac3267048c9034a35d2ac9acaad985633d6f891311fd4c6b69074227"
Oct 30 20:00:17.606209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.606149    4519 scope.go:122] "RemoveContainer" containerID="77c185db39abe14168ff210b1023b71adf73b9d9a52b20f090f3915c26b6a368"
Oct 30 20:00:17.606599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.606571    4519 scope.go:122] "RemoveContainer" containerID="5c353bd0cbbf14fec98e4f8e9b8a281950f406b4d072c4d0e10cdc7c4539c0e7"
Oct 30 20:00:17.606788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.606649    4519 scope.go:122] "RemoveContainer" containerID="16851ef75bb578136fee2e66dc11b39c5cb262602d7627177193d9b3197dffef"
Oct 30 20:00:17.814818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.814784    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerStarted","Data":"cfbc23064acf17599423fbbb78852ecee8cbec31d27cbb6a76a89894ea7c35f1"}
Oct 30 20:00:17.818778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.818745    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" event={"ID":"fef72380-91f7-4631-8100-4454165123fc","Type":"ContainerStarted","Data":"5c38944e8d0371af1f327ebef3601b2e75d553f3f0c463e49eef1b8e40df8990"}
Oct 30 20:00:17.820747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.820722    4519 generic.go:356] "Generic (PLEG): container finished" podID="07ec8783-2c39-4c35-836a-89a002783adb" containerID="4995a474cf2525222493adbd4d5c352b3e520e00bbece77ac6028cda5d819889" exitCode=60
Oct 30 20:00:17.820849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.820793    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" event={"ID":"07ec8783-2c39-4c35-836a-89a002783adb","Type":"ContainerDied","Data":"4995a474cf2525222493adbd4d5c352b3e520e00bbece77ac6028cda5d819889"}
Oct 30 20:00:17.820909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.820853    4519 scope.go:122] "RemoveContainer" containerID="47b62a84e9de3b829ae87d66ce873b17eace5ea957a2120b41358b0b73712a73"
Oct 30 20:00:17.821363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.821342    4519 scope.go:122] "RemoveContainer" containerID="4995a474cf2525222493adbd4d5c352b3e520e00bbece77ac6028cda5d819889"
Oct 30 20:00:17.821730 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:17.821701    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j_odh-ai-gateway-infra(07ec8783-2c39-4c35-836a-89a002783adb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" podUID="07ec8783-2c39-4c35-836a-89a002783adb"
Oct 30 20:00:17.858431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.858359    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"]
Oct 30 20:00:17.898231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.898199    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"]
Oct 30 20:00:17.947886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:17.939449    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"]
Oct 30 20:00:18.211310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.210902    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:18.266228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.266203    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:18.283330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.283299    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2324b1-81c5-46d8-adec-73e2aebbe111-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2324b1-81c5-46d8-adec-73e2aebbe111-maas-api-service-ca\") pod \"2d2324b1-81c5-46d8-adec-73e2aebbe111\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") "
Oct 30 20:00:18.283484 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.283383    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2324b1-81c5-46d8-adec-73e2aebbe111-kube-api-access-z2qz4\" (UniqueName: \"kubernetes.io/projected/2d2324b1-81c5-46d8-adec-73e2aebbe111-kube-api-access-z2qz4\") pod \"2d2324b1-81c5-46d8-adec-73e2aebbe111\" (UID: \"2d2324b1-81c5-46d8-adec-73e2aebbe111\") "
Oct 30 20:00:18.283813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.283787    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2324b1-81c5-46d8-adec-73e2aebbe111-maas-api-service-ca" pod "2d2324b1-81c5-46d8-adec-73e2aebbe111" (UID: "2d2324b1-81c5-46d8-adec-73e2aebbe111"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.284561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.284514    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d2324b1-81c5-46d8-adec-73e2aebbe111-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:18.285732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.285691    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2324b1-81c5-46d8-adec-73e2aebbe111-kube-api-access-z2qz4" pod "2d2324b1-81c5-46d8-adec-73e2aebbe111" (UID: "2d2324b1-81c5-46d8-adec-73e2aebbe111"). InnerVolumeSpecName "kube-api-access-z2qz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.385324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.385291    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61dc526f-7dee-49ac-b389-b340ceab3266-kube-api-access-ftg6v\" (UniqueName: \"kubernetes.io/projected/61dc526f-7dee-49ac-b389-b340ceab3266-kube-api-access-ftg6v\") pod \"61dc526f-7dee-49ac-b389-b340ceab3266\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") "
Oct 30 20:00:18.385595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.385344    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61dc526f-7dee-49ac-b389-b340ceab3266-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc526f-7dee-49ac-b389-b340ceab3266-maas-api-service-ca\") pod \"61dc526f-7dee-49ac-b389-b340ceab3266\" (UID: \"61dc526f-7dee-49ac-b389-b340ceab3266\") "
Oct 30 20:00:18.385831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.385804    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dc526f-7dee-49ac-b389-b340ceab3266-maas-api-service-ca" pod "61dc526f-7dee-49ac-b389-b340ceab3266" (UID: "61dc526f-7dee-49ac-b389-b340ceab3266"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.386055 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.386038    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc526f-7dee-49ac-b389-b340ceab3266-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:18.386112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.386056    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2qz4\" (UniqueName: \"kubernetes.io/projected/2d2324b1-81c5-46d8-adec-73e2aebbe111-kube-api-access-z2qz4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:18.387709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.387680    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dc526f-7dee-49ac-b389-b340ceab3266-kube-api-access-ftg6v" pod "61dc526f-7dee-49ac-b389-b340ceab3266" (UID: "61dc526f-7dee-49ac-b389-b340ceab3266"). InnerVolumeSpecName "kube-api-access-ftg6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.487861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.487824    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftg6v\" (UniqueName: \"kubernetes.io/projected/61dc526f-7dee-49ac-b389-b340ceab3266-kube-api-access-ftg6v\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:18.605914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.605876    4519 scope.go:122] "RemoveContainer" containerID="6cbc3690df74af6736f46bcd404467ccd5369b6c8451a052e285613ad4c2c52d"
Oct 30 20:00:18.606912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.606867    4519 scope.go:122] "RemoveContainer" containerID="734a84e2eabfffce1a57c7981c7db7e428cca6e4bf10fc1dece9f6a1483ac7bd"
Oct 30 20:00:18.832553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.832512    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" event={"ID":"d7d4bc1d-85a2-4553-942d-dbc161fde6b7","Type":"ContainerStarted","Data":"8757c64328b881e17e170d79a5586131350da8cefa0c3abec3c6a82a964c9426"}
Oct 30 20:00:18.834215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.834189    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"
Oct 30 20:00:18.834215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.834197    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8" event={"ID":"2d2324b1-81c5-46d8-adec-73e2aebbe111","Type":"ContainerDied","Data":"b41ef04ec7bd591643f46150849ed88055adb4a101669176677d8a9e1a400d48"}
Oct 30 20:00:18.834416 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.834257    4519 scope.go:122] "RemoveContainer" containerID="b12dceb40924d24979a596c3f5ea84d146b748f405dc29d27220d82eb6bce59c"
Oct 30 20:00:18.837020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.836970    4519 generic.go:356] "Generic (PLEG): container finished" podID="fef72380-91f7-4631-8100-4454165123fc" containerID="5c38944e8d0371af1f327ebef3601b2e75d553f3f0c463e49eef1b8e40df8990" exitCode=60
Oct 30 20:00:18.837141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.837081    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" event={"ID":"fef72380-91f7-4631-8100-4454165123fc","Type":"ContainerDied","Data":"5c38944e8d0371af1f327ebef3601b2e75d553f3f0c463e49eef1b8e40df8990"}
Oct 30 20:00:18.837630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.837603    4519 scope.go:122] "RemoveContainer" containerID="5c38944e8d0371af1f327ebef3601b2e75d553f3f0c463e49eef1b8e40df8990"
Oct 30 20:00:18.838092 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:18.838062    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr_odh-ai-gateway-infra(fef72380-91f7-4631-8100-4454165123fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" podUID="fef72380-91f7-4631-8100-4454165123fc"
Oct 30 20:00:18.843327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.843293    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerStarted","Data":"e0838616e03591319c6f94a62badf12e4d4d2721cad24d23215da2ae118c35b7"}
Oct 30 20:00:18.845169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.845143    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"
Oct 30 20:00:18.845298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.845173    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4" event={"ID":"61dc526f-7dee-49ac-b389-b340ceab3266","Type":"ContainerDied","Data":"585628331ebab008c7bbdb1b52a9349d40bbcbf5e58f64f4ca21b94d2260bd7d"}
Oct 30 20:00:18.851517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.849872    4519 generic.go:356] "Generic (PLEG): container finished" podID="32248e1e-4b52-4f73-b987-b0e348fab854" containerID="488a14fcb4522a00b37915ca2b051fdc2a66544d953a6e9970258a6f1ff5e9c0" exitCode=60
Oct 30 20:00:18.851517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.849926    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" event={"ID":"32248e1e-4b52-4f73-b987-b0e348fab854","Type":"ContainerDied","Data":"488a14fcb4522a00b37915ca2b051fdc2a66544d953a6e9970258a6f1ff5e9c0"}
Oct 30 20:00:18.851517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.850369    4519 scope.go:122] "RemoveContainer" containerID="488a14fcb4522a00b37915ca2b051fdc2a66544d953a6e9970258a6f1ff5e9c0"
Oct 30 20:00:18.851517 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:18.850708    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk_odh-ai-gateway-infra(32248e1e-4b52-4f73-b987-b0e348fab854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" podUID="32248e1e-4b52-4f73-b987-b0e348fab854"
Oct 30 20:00:18.860241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.860214    4519 scope.go:122] "RemoveContainer" containerID="cb922a9603074845b3017093cb16a53b15992610bdcb7ae665335e39278490a7"
Oct 30 20:00:18.863613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.862802    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" event={"ID":"c60757fe-86c3-4d66-a25c-e1ce2a276453","Type":"ContainerStarted","Data":"8ba2a55d46be6bbff035c73f7c667582edba08e80c16fb34e3a35620cf4228a7"}
Oct 30 20:00:18.866905 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.866881    4519 generic.go:356] "Generic (PLEG): container finished" podID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerID="cfbc23064acf17599423fbbb78852ecee8cbec31d27cbb6a76a89894ea7c35f1" exitCode=60
Oct 30 20:00:18.867163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.867141    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerDied","Data":"cfbc23064acf17599423fbbb78852ecee8cbec31d27cbb6a76a89894ea7c35f1"}
Oct 30 20:00:18.867758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.867739    4519 scope.go:122] "RemoveContainer" containerID="cfbc23064acf17599423fbbb78852ecee8cbec31d27cbb6a76a89894ea7c35f1"
Oct 30 20:00:18.868187 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:18.868147    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t_odh-ai-gateway-infra(33b31c7a-ff39-4203-9e69-e8b773b009e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4"
Oct 30 20:00:18.874729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.874707    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"]
Oct 30 20:00:18.923880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:18.923846    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"]
Oct 30 20:00:19.038899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.038720    4519 scope.go:122] "RemoveContainer" containerID="3e64ec9453bf1217038049bd8db1595f9e8e3e43e9415ff6804752d8e34255ab"
Oct 30 20:00:19.096386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.096307    4519 scope.go:122] "RemoveContainer" containerID="ed12adbaac3267048c9034a35d2ac9acaad985633d6f891311fd4c6b69074227"
Oct 30 20:00:19.096871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.096792    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:19.134018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.133997    4519 scope.go:122] "RemoveContainer" containerID="409b6990b0436f67a5e2f329aef167653c363a50267727dbd6dcacf177914f0a"
Oct 30 20:00:19.201484 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.201442    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ec8783-2c39-4c35-836a-89a002783adb-kube-api-access-vb6h6\" (UniqueName: \"kubernetes.io/projected/07ec8783-2c39-4c35-836a-89a002783adb-kube-api-access-vb6h6\") pod \"07ec8783-2c39-4c35-836a-89a002783adb\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") "
Oct 30 20:00:19.201673 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.201651    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ec8783-2c39-4c35-836a-89a002783adb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ec8783-2c39-4c35-836a-89a002783adb-maas-api-service-ca\") pod \"07ec8783-2c39-4c35-836a-89a002783adb\" (UID: \"07ec8783-2c39-4c35-836a-89a002783adb\") "
Oct 30 20:00:19.202038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.202013    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ec8783-2c39-4c35-836a-89a002783adb-maas-api-service-ca" pod "07ec8783-2c39-4c35-836a-89a002783adb" (UID: "07ec8783-2c39-4c35-836a-89a002783adb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.202409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.202390    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ec8783-2c39-4c35-836a-89a002783adb-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.203718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.203690    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ec8783-2c39-4c35-836a-89a002783adb-kube-api-access-vb6h6" pod "07ec8783-2c39-4c35-836a-89a002783adb" (UID: "07ec8783-2c39-4c35-836a-89a002783adb"). InnerVolumeSpecName "kube-api-access-vb6h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.213600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.213569    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"]
Oct 30 20:00:19.220854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.220834    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889840-tgqx8"]
Oct 30 20:00:19.322326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.322236    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6h6\" (UniqueName: \"kubernetes.io/projected/07ec8783-2c39-4c35-836a-89a002783adb-kube-api-access-vb6h6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.610306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.608739    4519 scope.go:122] "RemoveContainer" containerID="16520161fef153a49672fa9211a3d8975fcec90cd0d5c004f2645e93b1ca3bed"
Oct 30 20:00:19.617269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.616993    4519 scope.go:122] "RemoveContainer" containerID="b4c56e9b9c92d966626d80faa7342603fd8c5c944a4709a3226915be8a09aebc"
Oct 30 20:00:19.679188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.679123    4519 scope.go:122] "RemoveContainer" containerID="b3805b0bd337ef8599681a15e449978cf190828644cb338c2256c0a2b167e1d4"
Oct 30 20:00:19.685153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.684716    4519 scope.go:122] "RemoveContainer" containerID="9c72c9362c93d64da03a6f7827b26c9a6ca22d1a9640812ca9ca612ac390bfa2"
Oct 30 20:00:19.690494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.689245    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" path="/var/lib/kubelet/pods/2d2324b1-81c5-46d8-adec-73e2aebbe111/volumes"
Oct 30 20:00:19.710388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.710358    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"]
Oct 30 20:00:19.724177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.722984    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889840-8t5h4"]
Oct 30 20:00:19.891093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.891008    4519 generic.go:356] "Generic (PLEG): container finished" podID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerID="8757c64328b881e17e170d79a5586131350da8cefa0c3abec3c6a82a964c9426" exitCode=60
Oct 30 20:00:19.891660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.891082    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" event={"ID":"d7d4bc1d-85a2-4553-942d-dbc161fde6b7","Type":"ContainerDied","Data":"8757c64328b881e17e170d79a5586131350da8cefa0c3abec3c6a82a964c9426"}
Oct 30 20:00:19.891660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.891185    4519 scope.go:122] "RemoveContainer" containerID="5c353bd0cbbf14fec98e4f8e9b8a281950f406b4d072c4d0e10cdc7c4539c0e7"
Oct 30 20:00:19.892066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.891663    4519 scope.go:122] "RemoveContainer" containerID="8757c64328b881e17e170d79a5586131350da8cefa0c3abec3c6a82a964c9426"
Oct 30 20:00:19.892124 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:19.892062    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx_odh-ai-gateway-infra(d7d4bc1d-85a2-4553-942d-dbc161fde6b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7"
Oct 30 20:00:19.892932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.892906    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"]
Oct 30 20:00:19.899062 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.898293    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j" event={"ID":"07ec8783-2c39-4c35-836a-89a002783adb","Type":"ContainerDied","Data":"230981213c60866d585b628bd83bd40d7cc2b65f9c1994f6633157879162c551"}
Oct 30 20:00:19.899062 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.898403    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"
Oct 30 20:00:19.906520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.906494    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerStarted","Data":"14b873f5b6e705426b1df317ae150f87e89c4983e12794aa5775889b6e7bc219"}
Oct 30 20:00:19.908809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.908783    4519 generic.go:356] "Generic (PLEG): container finished" podID="906f4700-7e26-4304-be69-bd6c77566780" containerID="e0838616e03591319c6f94a62badf12e4d4d2721cad24d23215da2ae118c35b7" exitCode=60
Oct 30 20:00:19.908901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.908847    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerDied","Data":"e0838616e03591319c6f94a62badf12e4d4d2721cad24d23215da2ae118c35b7"}
Oct 30 20:00:19.909193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.909179    4519 scope.go:122] "RemoveContainer" containerID="e0838616e03591319c6f94a62badf12e4d4d2721cad24d23215da2ae118c35b7"
Oct 30 20:00:19.909545 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:19.909524    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx_odh-ai-gateway-infra(906f4700-7e26-4304-be69-bd6c77566780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" podUID="906f4700-7e26-4304-be69-bd6c77566780"
Oct 30 20:00:19.913771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.913737    4519 generic.go:356] "Generic (PLEG): container finished" podID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerID="8ba2a55d46be6bbff035c73f7c667582edba08e80c16fb34e3a35620cf4228a7" exitCode=60
Oct 30 20:00:19.913869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.913832    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" event={"ID":"c60757fe-86c3-4d66-a25c-e1ce2a276453","Type":"ContainerDied","Data":"8ba2a55d46be6bbff035c73f7c667582edba08e80c16fb34e3a35620cf4228a7"}
Oct 30 20:00:19.914217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.914194    4519 scope.go:122] "RemoveContainer" containerID="8ba2a55d46be6bbff035c73f7c667582edba08e80c16fb34e3a35620cf4228a7"
Oct 30 20:00:19.914456 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:19.914434    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb_odh-ai-gateway-infra(c60757fe-86c3-4d66-a25c-e1ce2a276453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453"
Oct 30 20:00:19.916044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.916023    4519 generic.go:356] "Generic (PLEG): container finished" podID="032754c4-d986-4767-bc09-19ff1ba923a7" containerID="c9add0aa4ba2f21c6ab83f88ad1a1db6b53c9d713cffd3b54e440dfbb4a54a1a" exitCode=60
Oct 30 20:00:19.916148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.916100    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" event={"ID":"032754c4-d986-4767-bc09-19ff1ba923a7","Type":"ContainerDied","Data":"c9add0aa4ba2f21c6ab83f88ad1a1db6b53c9d713cffd3b54e440dfbb4a54a1a"}
Oct 30 20:00:19.916417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.916394    4519 scope.go:122] "RemoveContainer" containerID="c9add0aa4ba2f21c6ab83f88ad1a1db6b53c9d713cffd3b54e440dfbb4a54a1a"
Oct 30 20:00:19.916714 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:19.916686    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d_odh-ai-gateway-infra(032754c4-d986-4767-bc09-19ff1ba923a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" podUID="032754c4-d986-4767-bc09-19ff1ba923a7"
Oct 30 20:00:19.945184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:19.945158    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"]
Oct 30 20:00:20.035485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.035442    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"]
Oct 30 20:00:20.146068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.146034    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"]
Oct 30 20:00:20.186175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.184414    4519 scope.go:122] "RemoveContainer" containerID="4995a474cf2525222493adbd4d5c352b3e520e00bbece77ac6028cda5d819889"
Oct 30 20:00:20.206350 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.206330    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:20.207363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.207345    4519 scope.go:122] "RemoveContainer" containerID="16851ef75bb578136fee2e66dc11b39c5cb262602d7627177193d9b3197dffef"
Oct 30 20:00:20.222587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.222566    4519 scope.go:122] "RemoveContainer" containerID="77c185db39abe14168ff210b1023b71adf73b9d9a52b20f090f3915c26b6a368"
Oct 30 20:00:20.240207 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.240183    4519 scope.go:122] "RemoveContainer" containerID="734a84e2eabfffce1a57c7981c7db7e428cca6e4bf10fc1dece9f6a1483ac7bd"
Oct 30 20:00:20.268482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.268394    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"]
Oct 30 20:00:20.277676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.277642    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889840-s5f5j"]
Oct 30 20:00:20.396118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.395759    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32248e1e-4b52-4f73-b987-b0e348fab854-kube-api-access-zmrpq\" (UniqueName: \"kubernetes.io/projected/32248e1e-4b52-4f73-b987-b0e348fab854-kube-api-access-zmrpq\") pod \"32248e1e-4b52-4f73-b987-b0e348fab854\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") "
Oct 30 20:00:20.396118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.395884    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32248e1e-4b52-4f73-b987-b0e348fab854-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32248e1e-4b52-4f73-b987-b0e348fab854-maas-api-service-ca\") pod \"32248e1e-4b52-4f73-b987-b0e348fab854\" (UID: \"32248e1e-4b52-4f73-b987-b0e348fab854\") "
Oct 30 20:00:20.396858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.396809    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32248e1e-4b52-4f73-b987-b0e348fab854-maas-api-service-ca" pod "32248e1e-4b52-4f73-b987-b0e348fab854" (UID: "32248e1e-4b52-4f73-b987-b0e348fab854"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:20.406700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.406662    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32248e1e-4b52-4f73-b987-b0e348fab854-kube-api-access-zmrpq" pod "32248e1e-4b52-4f73-b987-b0e348fab854" (UID: "32248e1e-4b52-4f73-b987-b0e348fab854"). InnerVolumeSpecName "kube-api-access-zmrpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.427131 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.426987    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:20.435481 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.435443    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:20.498909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.498872    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrpq\" (UniqueName: \"kubernetes.io/projected/32248e1e-4b52-4f73-b987-b0e348fab854-kube-api-access-zmrpq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.498909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.498911    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32248e1e-4b52-4f73-b987-b0e348fab854-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.599832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.599796    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef72380-91f7-4631-8100-4454165123fc-kube-api-access-57zsk\" (UniqueName: \"kubernetes.io/projected/fef72380-91f7-4631-8100-4454165123fc-kube-api-access-57zsk\") pod \"fef72380-91f7-4631-8100-4454165123fc\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") "
Oct 30 20:00:20.600115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.600099    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b31c7a-ff39-4203-9e69-e8b773b009e4-kube-api-access-nt4gj\" (UniqueName: \"kubernetes.io/projected/33b31c7a-ff39-4203-9e69-e8b773b009e4-kube-api-access-nt4gj\") pod \"33b31c7a-ff39-4203-9e69-e8b773b009e4\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") "
Oct 30 20:00:20.600164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.600149    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef72380-91f7-4631-8100-4454165123fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef72380-91f7-4631-8100-4454165123fc-maas-api-service-ca\") pod \"fef72380-91f7-4631-8100-4454165123fc\" (UID: \"fef72380-91f7-4631-8100-4454165123fc\") "
Oct 30 20:00:20.600232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.600219    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b31c7a-ff39-4203-9e69-e8b773b009e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b31c7a-ff39-4203-9e69-e8b773b009e4-maas-api-service-ca\") pod \"33b31c7a-ff39-4203-9e69-e8b773b009e4\" (UID: \"33b31c7a-ff39-4203-9e69-e8b773b009e4\") "
Oct 30 20:00:20.600704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.600590    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef72380-91f7-4631-8100-4454165123fc-maas-api-service-ca" pod "fef72380-91f7-4631-8100-4454165123fc" (UID: "fef72380-91f7-4631-8100-4454165123fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:20.600861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.600698    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b31c7a-ff39-4203-9e69-e8b773b009e4-maas-api-service-ca" pod "33b31c7a-ff39-4203-9e69-e8b773b009e4" (UID: "33b31c7a-ff39-4203-9e69-e8b773b009e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:20.601084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.601057    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef72380-91f7-4631-8100-4454165123fc-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.601184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.601090    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b31c7a-ff39-4203-9e69-e8b773b009e4-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.602375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.602346    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef72380-91f7-4631-8100-4454165123fc-kube-api-access-57zsk" pod "fef72380-91f7-4631-8100-4454165123fc" (UID: "fef72380-91f7-4631-8100-4454165123fc"). InnerVolumeSpecName "kube-api-access-57zsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.602570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.602398    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b31c7a-ff39-4203-9e69-e8b773b009e4-kube-api-access-nt4gj" pod "33b31c7a-ff39-4203-9e69-e8b773b009e4" (UID: "33b31c7a-ff39-4203-9e69-e8b773b009e4"). InnerVolumeSpecName "kube-api-access-nt4gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.606071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.606045    4519 scope.go:122] "RemoveContainer" containerID="7d7c3002113dedbe35ac831c09266928ab5962a68b032d1c2f00b595cac41a01"
Oct 30 20:00:20.702865 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.702829    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4gj\" (UniqueName: \"kubernetes.io/projected/33b31c7a-ff39-4203-9e69-e8b773b009e4-kube-api-access-nt4gj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.702865 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.702865    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57zsk\" (UniqueName: \"kubernetes.io/projected/fef72380-91f7-4631-8100-4454165123fc-kube-api-access-57zsk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.932945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.932910    4519 generic.go:356] "Generic (PLEG): container finished" podID="64d4d50d-febd-43fe-90dc-cb8836687123" containerID="14b873f5b6e705426b1df317ae150f87e89c4983e12794aa5775889b6e7bc219" exitCode=60
Oct 30 20:00:20.933387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.932986    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerDied","Data":"14b873f5b6e705426b1df317ae150f87e89c4983e12794aa5775889b6e7bc219"}
Oct 30 20:00:20.933387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.933059    4519 scope.go:122] "RemoveContainer" containerID="6cbc3690df74af6736f46bcd404467ccd5369b6c8451a052e285613ad4c2c52d"
Oct 30 20:00:20.933725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.933706    4519 scope.go:122] "RemoveContainer" containerID="14b873f5b6e705426b1df317ae150f87e89c4983e12794aa5775889b6e7bc219"
Oct 30 20:00:20.934189 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:20.934161    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6_odh-ai-gateway-infra(64d4d50d-febd-43fe-90dc-cb8836687123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" podUID="64d4d50d-febd-43fe-90dc-cb8836687123"
Oct 30 20:00:20.937264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.937246    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"
Oct 30 20:00:20.937403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.937243    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk" event={"ID":"32248e1e-4b52-4f73-b987-b0e348fab854","Type":"ContainerDied","Data":"fdbd0b76f81503eec0be171def539a04e0da5a370ccf1e3f924c53f1b3b61a76"}
Oct 30 20:00:20.941794 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.940922    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerStarted","Data":"4bd39529858b48392b3b1919fce19381d9dd3921ca036186d250425d691efb4e"}
Oct 30 20:00:20.961319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.959902    4519 generic.go:356] "Generic (PLEG): container finished" podID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerID="9295cca89a7e29a0f5058391d99ad10d91ac94f428eced9aafcf393d191317c0" exitCode=60
Oct 30 20:00:20.961319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.960024    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" event={"ID":"e6456596-13e7-48e2-af8b-1ec739ca9871","Type":"ContainerDied","Data":"9295cca89a7e29a0f5058391d99ad10d91ac94f428eced9aafcf393d191317c0"}
Oct 30 20:00:20.961319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.960495    4519 scope.go:122] "RemoveContainer" containerID="9295cca89a7e29a0f5058391d99ad10d91ac94f428eced9aafcf393d191317c0"
Oct 30 20:00:20.961319 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:20.960941    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv_odh-ai-gateway-infra(e6456596-13e7-48e2-af8b-1ec739ca9871)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871"
Oct 30 20:00:20.968198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.968082    4519 scope.go:122] "RemoveContainer" containerID="488a14fcb4522a00b37915ca2b051fdc2a66544d953a6e9970258a6f1ff5e9c0"
Oct 30 20:00:20.969056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.969029    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerStarted","Data":"da93cfda902a96da7e6dfdfccd9d8444197cbb557bee8dfa5613b8528e524184"}
Oct 30 20:00:20.972419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:20.972393    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerStarted","Data":"42bbe6c830c5bd51e0193f1962af49158de9d4b9977d0ff970438105783bb5a7"}
Oct 30 20:00:21.003139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.003098    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t" event={"ID":"33b31c7a-ff39-4203-9e69-e8b773b009e4","Type":"ContainerDied","Data":"fce41aad16dfeafb0fefcfef9bc20df71d68d8897bb4c80afb8d953949ba6c2d"}
Oct 30 20:00:21.003495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.003459    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"
Oct 30 20:00:21.027034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.027011    4519 scope.go:122] "RemoveContainer" containerID="16520161fef153a49672fa9211a3d8975fcec90cd0d5c004f2645e93b1ca3bed"
Oct 30 20:00:21.028253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.028229    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr" event={"ID":"fef72380-91f7-4631-8100-4454165123fc","Type":"ContainerDied","Data":"ca96375b8630b42db5231c1ca661105b1473f106530bd8151e846daa22cbe360"}
Oct 30 20:00:21.030746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.028530    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"
Oct 30 20:00:21.084249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.084222    4519 scope.go:122] "RemoveContainer" containerID="cfbc23064acf17599423fbbb78852ecee8cbec31d27cbb6a76a89894ea7c35f1"
Oct 30 20:00:21.216761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.216730    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"]
Oct 30 20:00:21.269508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.267560    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"]
Oct 30 20:00:21.274574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.274486    4519 scope.go:122] "RemoveContainer" containerID="5c38944e8d0371af1f327ebef3601b2e75d553f3f0c463e49eef1b8e40df8990"
Oct 30 20:00:21.276182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.276156    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-21c4ab-29889840-h4wpk"]
Oct 30 20:00:21.282532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.282507    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"]
Oct 30 20:00:21.285399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.285368    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:21.291252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.291222    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:21.388685 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.388640    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"]
Oct 30 20:00:21.405860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.405824    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0506c9-29889840-6pwsr"]
Oct 30 20:00:21.409907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.409316    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:21.454457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.454356    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60757fe-86c3-4d66-a25c-e1ce2a276453-kube-api-access-5sbtf\" (UniqueName: \"kubernetes.io/projected/c60757fe-86c3-4d66-a25c-e1ce2a276453-kube-api-access-5sbtf\") pod \"c60757fe-86c3-4d66-a25c-e1ce2a276453\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") "
Oct 30 20:00:21.454700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.454597    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906f4700-7e26-4304-be69-bd6c77566780-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f4700-7e26-4304-be69-bd6c77566780-maas-api-service-ca\") pod \"906f4700-7e26-4304-be69-bd6c77566780\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") "
Oct 30 20:00:21.455182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.455156    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906f4700-7e26-4304-be69-bd6c77566780-maas-api-service-ca" pod "906f4700-7e26-4304-be69-bd6c77566780" (UID: "906f4700-7e26-4304-be69-bd6c77566780"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.457569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.456979    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906f4700-7e26-4304-be69-bd6c77566780-kube-api-access-nsd69\" (UniqueName: \"kubernetes.io/projected/906f4700-7e26-4304-be69-bd6c77566780-kube-api-access-nsd69\") pod \"906f4700-7e26-4304-be69-bd6c77566780\" (UID: \"906f4700-7e26-4304-be69-bd6c77566780\") "
Oct 30 20:00:21.457569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.457128    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60757fe-86c3-4d66-a25c-e1ce2a276453-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60757fe-86c3-4d66-a25c-e1ce2a276453-maas-api-service-ca\") pod \"c60757fe-86c3-4d66-a25c-e1ce2a276453\" (UID: \"c60757fe-86c3-4d66-a25c-e1ce2a276453\") "
Oct 30 20:00:21.457569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.457535    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60757fe-86c3-4d66-a25c-e1ce2a276453-maas-api-service-ca" pod "c60757fe-86c3-4d66-a25c-e1ce2a276453" (UID: "c60757fe-86c3-4d66-a25c-e1ce2a276453"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.458249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.458222    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c60757fe-86c3-4d66-a25c-e1ce2a276453-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.458355 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.458257    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f4700-7e26-4304-be69-bd6c77566780-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.458788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.458763    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60757fe-86c3-4d66-a25c-e1ce2a276453-kube-api-access-5sbtf" pod "c60757fe-86c3-4d66-a25c-e1ce2a276453" (UID: "c60757fe-86c3-4d66-a25c-e1ce2a276453"). InnerVolumeSpecName "kube-api-access-5sbtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.460456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.460431    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906f4700-7e26-4304-be69-bd6c77566780-kube-api-access-nsd69" pod "906f4700-7e26-4304-be69-bd6c77566780" (UID: "906f4700-7e26-4304-be69-bd6c77566780"). InnerVolumeSpecName "kube-api-access-nsd69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.491810 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.491782    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:21.518295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.518251    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"]
Oct 30 20:00:21.523513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.523476    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889840-lbs8t"]
Oct 30 20:00:21.559095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.559062    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032754c4-d986-4767-bc09-19ff1ba923a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032754c4-d986-4767-bc09-19ff1ba923a7-maas-api-service-ca\") pod \"032754c4-d986-4767-bc09-19ff1ba923a7\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") "
Oct 30 20:00:21.559450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.559428    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032754c4-d986-4767-bc09-19ff1ba923a7-kube-api-access-k9rrf\" (UniqueName: \"kubernetes.io/projected/032754c4-d986-4767-bc09-19ff1ba923a7-kube-api-access-k9rrf\") pod \"032754c4-d986-4767-bc09-19ff1ba923a7\" (UID: \"032754c4-d986-4767-bc09-19ff1ba923a7\") "
Oct 30 20:00:21.560096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.560070    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-maas-api-service-ca\") pod \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") "
Oct 30 20:00:21.560397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.560377    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-kube-api-access-7rwh6\" (UniqueName: \"kubernetes.io/projected/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-kube-api-access-7rwh6\") pod \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\" (UID: \"d7d4bc1d-85a2-4553-942d-dbc161fde6b7\") "
Oct 30 20:00:21.560962 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.560933    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-maas-api-service-ca" pod "d7d4bc1d-85a2-4553-942d-dbc161fde6b7" (UID: "d7d4bc1d-85a2-4553-942d-dbc161fde6b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.561209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.561188    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsd69\" (UniqueName: \"kubernetes.io/projected/906f4700-7e26-4304-be69-bd6c77566780-kube-api-access-nsd69\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.561286 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.561215    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbtf\" (UniqueName: \"kubernetes.io/projected/c60757fe-86c3-4d66-a25c-e1ce2a276453-kube-api-access-5sbtf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.561286 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.561230    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.561648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.561610    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032754c4-d986-4767-bc09-19ff1ba923a7-maas-api-service-ca" pod "032754c4-d986-4767-bc09-19ff1ba923a7" (UID: "032754c4-d986-4767-bc09-19ff1ba923a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.563324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.563299    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032754c4-d986-4767-bc09-19ff1ba923a7-kube-api-access-k9rrf" pod "032754c4-d986-4767-bc09-19ff1ba923a7" (UID: "032754c4-d986-4767-bc09-19ff1ba923a7"). InnerVolumeSpecName "kube-api-access-k9rrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.565318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.565298    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-kube-api-access-7rwh6" pod "d7d4bc1d-85a2-4553-942d-dbc161fde6b7" (UID: "d7d4bc1d-85a2-4553-942d-dbc161fde6b7"). InnerVolumeSpecName "kube-api-access-7rwh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.615881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.615851    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ec8783-2c39-4c35-836a-89a002783adb" path="/var/lib/kubelet/pods/07ec8783-2c39-4c35-836a-89a002783adb/volumes"
Oct 30 20:00:21.616373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.616359    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" path="/var/lib/kubelet/pods/32248e1e-4b52-4f73-b987-b0e348fab854/volumes"
Oct 30 20:00:21.616852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.616837    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" path="/var/lib/kubelet/pods/33b31c7a-ff39-4203-9e69-e8b773b009e4/volumes"
Oct 30 20:00:21.617590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.617574    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" path="/var/lib/kubelet/pods/61dc526f-7dee-49ac-b389-b340ceab3266/volumes"
Oct 30 20:00:21.618003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.617990    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef72380-91f7-4631-8100-4454165123fc" path="/var/lib/kubelet/pods/fef72380-91f7-4631-8100-4454165123fc/volumes"
Oct 30 20:00:21.663282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.663236    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032754c4-d986-4767-bc09-19ff1ba923a7-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.663282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.663276    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9rrf\" (UniqueName: \"kubernetes.io/projected/032754c4-d986-4767-bc09-19ff1ba923a7-kube-api-access-k9rrf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.663515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:21.663293    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwh6\" (UniqueName: \"kubernetes.io/projected/d7d4bc1d-85a2-4553-942d-dbc161fde6b7-kube-api-access-7rwh6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:22.044555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.044515    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"
Oct 30 20:00:22.044976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.044519    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx" event={"ID":"906f4700-7e26-4304-be69-bd6c77566780","Type":"ContainerDied","Data":"119a70806baba014a7479d4a21da8fafb2bede2961af2b5abda1b80d79ed90b5"}
Oct 30 20:00:22.044976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.044708    4519 scope.go:122] "RemoveContainer" containerID="e0838616e03591319c6f94a62badf12e4d4d2721cad24d23215da2ae118c35b7"
Oct 30 20:00:22.049211 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.049177    4519 generic.go:356] "Generic (PLEG): container finished" podID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerID="4bd39529858b48392b3b1919fce19381d9dd3921ca036186d250425d691efb4e" exitCode=60
Oct 30 20:00:22.049331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.049258    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerDied","Data":"4bd39529858b48392b3b1919fce19381d9dd3921ca036186d250425d691efb4e"}
Oct 30 20:00:22.049855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.049835    4519 scope.go:122] "RemoveContainer" containerID="4bd39529858b48392b3b1919fce19381d9dd3921ca036186d250425d691efb4e"
Oct 30 20:00:22.050573 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:22.050546    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc_odh-ai-gateway-infra(32cb5bde-e08d-496d-a83b-365d19d8d4a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4"
Oct 30 20:00:22.052189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.051478    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"
Oct 30 20:00:22.052189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.052099    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb" event={"ID":"c60757fe-86c3-4d66-a25c-e1ce2a276453","Type":"ContainerDied","Data":"dee500290c8b1a1e35854ee789bd19d12449d03b38c10e73d1c3eb8ee8b49816"}
Oct 30 20:00:22.055780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.055132    4519 generic.go:356] "Generic (PLEG): container finished" podID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerID="a3cfb109407e4c0cd8de13189da3803105768d22e1675f4e3118fffd1789ab02" exitCode=60
Oct 30 20:00:22.055780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.055205    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" event={"ID":"badf3d12-c146-4ab3-b2b7-3c2fc1af5574","Type":"ContainerDied","Data":"a3cfb109407e4c0cd8de13189da3803105768d22e1675f4e3118fffd1789ab02"}
Oct 30 20:00:22.056381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.056350    4519 scope.go:122] "RemoveContainer" containerID="a3cfb109407e4c0cd8de13189da3803105768d22e1675f4e3118fffd1789ab02"
Oct 30 20:00:22.058430 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:22.056786    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-bbbl2_odh-ai-gateway-infra(badf3d12-c146-4ab3-b2b7-3c2fc1af5574)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574"
Oct 30 20:00:22.059344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.058655    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d" event={"ID":"032754c4-d986-4767-bc09-19ff1ba923a7","Type":"ContainerDied","Data":"d430c3e956790c93ca707109252174f0745624f6da04bc45a475d9038f7bf880"}
Oct 30 20:00:22.059344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.058754    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"
Oct 30 20:00:22.060892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.060804    4519 scope.go:122] "RemoveContainer" containerID="9c72c9362c93d64da03a6f7827b26c9a6ca22d1a9640812ca9ca612ac390bfa2"
Oct 30 20:00:22.065510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.065478    4519 generic.go:356] "Generic (PLEG): container finished" podID="a278a416-54cd-4256-ac17-6e23b31c0671" containerID="da93cfda902a96da7e6dfdfccd9d8444197cbb557bee8dfa5613b8528e524184" exitCode=60
Oct 30 20:00:22.065510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.065499    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerDied","Data":"da93cfda902a96da7e6dfdfccd9d8444197cbb557bee8dfa5613b8528e524184"}
Oct 30 20:00:22.065939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.065922    4519 scope.go:122] "RemoveContainer" containerID="da93cfda902a96da7e6dfdfccd9d8444197cbb557bee8dfa5613b8528e524184"
Oct 30 20:00:22.066265 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:22.066243    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5_odh-ai-gateway-infra(a278a416-54cd-4256-ac17-6e23b31c0671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" podUID="a278a416-54cd-4256-ac17-6e23b31c0671"
Oct 30 20:00:22.071105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.069318    4519 generic.go:356] "Generic (PLEG): container finished" podID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerID="42bbe6c830c5bd51e0193f1962af49158de9d4b9977d0ff970438105783bb5a7" exitCode=60
Oct 30 20:00:22.071105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.069420    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerDied","Data":"42bbe6c830c5bd51e0193f1962af49158de9d4b9977d0ff970438105783bb5a7"}
Oct 30 20:00:22.071105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.069875    4519 scope.go:122] "RemoveContainer" containerID="42bbe6c830c5bd51e0193f1962af49158de9d4b9977d0ff970438105783bb5a7"
Oct 30 20:00:22.071105 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:22.070211    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k_odh-ai-gateway-infra(2ea9d0ec-fb50-4c79-b2d0-1ec024506b59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59"
Oct 30 20:00:22.074091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.074068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx" event={"ID":"d7d4bc1d-85a2-4553-942d-dbc161fde6b7","Type":"ContainerDied","Data":"1ade0933ab6653cad0977fe5cbc137535c95abd192efe1a8218dab7760910d86"}
Oct 30 20:00:22.074288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.074274    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"
Oct 30 20:00:22.083863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.083839    4519 scope.go:122] "RemoveContainer" containerID="8ba2a55d46be6bbff035c73f7c667582edba08e80c16fb34e3a35620cf4228a7"
Oct 30 20:00:22.085039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.085012    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"]
Oct 30 20:00:22.103006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.102984    4519 scope.go:122] "RemoveContainer" containerID="7d7c3002113dedbe35ac831c09266928ab5962a68b032d1c2f00b595cac41a01"
Oct 30 20:00:22.138561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.138524    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"]
Oct 30 20:00:22.162504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.143237    4519 scope.go:122] "RemoveContainer" containerID="c9add0aa4ba2f21c6ab83f88ad1a1db6b53c9d713cffd3b54e440dfbb4a54a1a"
Oct 30 20:00:22.162504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.146841    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"]
Oct 30 20:00:22.162504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.157689    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb16ce-29889840-h2lwx"]
Oct 30 20:00:22.216476 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.216430    4519 scope.go:122] "RemoveContainer" containerID="b3805b0bd337ef8599681a15e449978cf190828644cb338c2256c0a2b167e1d4"
Oct 30 20:00:22.218782 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.218761    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"]
Oct 30 20:00:22.262283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.262258    4519 scope.go:122] "RemoveContainer" containerID="b4c56e9b9c92d966626d80faa7342603fd8c5c944a4709a3226915be8a09aebc"
Oct 30 20:00:22.266076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.266051    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:22.278242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.278215    4519 scope.go:122] "RemoveContainer" containerID="8757c64328b881e17e170d79a5586131350da8cefa0c3abec3c6a82a964c9426"
Oct 30 20:00:22.286726 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.286693    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"]
Oct 30 20:00:22.292494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.292450    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889840-5knfx"]
Oct 30 20:00:22.332512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.332478    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"]
Oct 30 20:00:22.369387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.369349    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"]
Oct 30 20:00:22.375256 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.375209    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889840-fzbcb"]
Oct 30 20:00:22.398378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.398350    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d4d50d-febd-43fe-90dc-cb8836687123-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d4d50d-febd-43fe-90dc-cb8836687123-maas-api-service-ca\") pod \"64d4d50d-febd-43fe-90dc-cb8836687123\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") "
Oct 30 20:00:22.398651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.398634    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d4d50d-febd-43fe-90dc-cb8836687123-kube-api-access-2vsgs\" (UniqueName: \"kubernetes.io/projected/64d4d50d-febd-43fe-90dc-cb8836687123-kube-api-access-2vsgs\") pod \"64d4d50d-febd-43fe-90dc-cb8836687123\" (UID: \"64d4d50d-febd-43fe-90dc-cb8836687123\") "
Oct 30 20:00:22.398770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.398740    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d4d50d-febd-43fe-90dc-cb8836687123-maas-api-service-ca" pod "64d4d50d-febd-43fe-90dc-cb8836687123" (UID: "64d4d50d-febd-43fe-90dc-cb8836687123"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:22.399070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.399053    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d4d50d-febd-43fe-90dc-cb8836687123-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:22.399951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.399921    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"]
Oct 30 20:00:22.401121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.401095    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d4d50d-febd-43fe-90dc-cb8836687123-kube-api-access-2vsgs" pod "64d4d50d-febd-43fe-90dc-cb8836687123" (UID: "64d4d50d-febd-43fe-90dc-cb8836687123"). InnerVolumeSpecName "kube-api-access-2vsgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:22.405376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.405354    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-aa42c3-29889840-kzp2d"]
Oct 30 20:00:22.499621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.499578    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:22.500338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.500314    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vsgs\" (UniqueName: \"kubernetes.io/projected/64d4d50d-febd-43fe-90dc-cb8836687123-kube-api-access-2vsgs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:22.601870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.601795    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6456596-13e7-48e2-af8b-1ec739ca9871-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6456596-13e7-48e2-af8b-1ec739ca9871-maas-api-service-ca\") pod \"e6456596-13e7-48e2-af8b-1ec739ca9871\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") "
Oct 30 20:00:22.602013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.601879    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6456596-13e7-48e2-af8b-1ec739ca9871-kube-api-access-shwwz\" (UniqueName: \"kubernetes.io/projected/e6456596-13e7-48e2-af8b-1ec739ca9871-kube-api-access-shwwz\") pod \"e6456596-13e7-48e2-af8b-1ec739ca9871\" (UID: \"e6456596-13e7-48e2-af8b-1ec739ca9871\") "
Oct 30 20:00:22.602183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.602159    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6456596-13e7-48e2-af8b-1ec739ca9871-maas-api-service-ca" pod "e6456596-13e7-48e2-af8b-1ec739ca9871" (UID: "e6456596-13e7-48e2-af8b-1ec739ca9871"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:22.602577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.602554    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6456596-13e7-48e2-af8b-1ec739ca9871-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:22.604284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.604259    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6456596-13e7-48e2-af8b-1ec739ca9871-kube-api-access-shwwz" pod "e6456596-13e7-48e2-af8b-1ec739ca9871" (UID: "e6456596-13e7-48e2-af8b-1ec739ca9871"). InnerVolumeSpecName "kube-api-access-shwwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:22.705056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:22.705020    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwwz\" (UniqueName: \"kubernetes.io/projected/e6456596-13e7-48e2-af8b-1ec739ca9871-kube-api-access-shwwz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.097752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.097718    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"
Oct 30 20:00:23.098220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.097728    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6" event={"ID":"64d4d50d-febd-43fe-90dc-cb8836687123","Type":"ContainerDied","Data":"80b46487f6aea43c8f51e0265987167ff7a13590ccd8c54f922ff5f4409d84ea"}
Oct 30 20:00:23.098220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.097866    4519 scope.go:122] "RemoveContainer" containerID="14b873f5b6e705426b1df317ae150f87e89c4983e12794aa5775889b6e7bc219"
Oct 30 20:00:23.114841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.114812    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"
Oct 30 20:00:23.115367 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.115334    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv" event={"ID":"e6456596-13e7-48e2-af8b-1ec739ca9871","Type":"ContainerDied","Data":"b00e154c9713ce22d19f40fcd300757cde3283b67432e6de1b4930124013efdd"}
Oct 30 20:00:23.115514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.115402    4519 scope.go:122] "RemoveContainer" containerID="9295cca89a7e29a0f5058391d99ad10d91ac94f428eced9aafcf393d191317c0"
Oct 30 20:00:23.202408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.202371    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"]
Oct 30 20:00:23.212733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.212693    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-93263c-29889840-8vvx6"]
Oct 30 20:00:23.323934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.323894    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"]
Oct 30 20:00:23.332706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.332672    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889840-58wjv"]
Oct 30 20:00:23.371107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.371037    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:23.416297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.416267    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cb5bde-e08d-496d-a83b-365d19d8d4a4-kube-api-access-sk7g4\" (UniqueName: \"kubernetes.io/projected/32cb5bde-e08d-496d-a83b-365d19d8d4a4-kube-api-access-sk7g4\") pod \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") "
Oct 30 20:00:23.416626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.416597    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cb5bde-e08d-496d-a83b-365d19d8d4a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cb5bde-e08d-496d-a83b-365d19d8d4a4-maas-api-service-ca\") pod \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\" (UID: \"32cb5bde-e08d-496d-a83b-365d19d8d4a4\") "
Oct 30 20:00:23.417225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.417195    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cb5bde-e08d-496d-a83b-365d19d8d4a4-maas-api-service-ca" pod "32cb5bde-e08d-496d-a83b-365d19d8d4a4" (UID: "32cb5bde-e08d-496d-a83b-365d19d8d4a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.418023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.417793    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cb5bde-e08d-496d-a83b-365d19d8d4a4-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.419784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.419753    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cb5bde-e08d-496d-a83b-365d19d8d4a4-kube-api-access-sk7g4" pod "32cb5bde-e08d-496d-a83b-365d19d8d4a4" (UID: "32cb5bde-e08d-496d-a83b-365d19d8d4a4"). InnerVolumeSpecName "kube-api-access-sk7g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.428758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.428722    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:23.430943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.430898    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:23.450311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.450286    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"]
Oct 30 20:00:23.520440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.520392    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-kube-api-access-8m5sx\" (UniqueName: \"kubernetes.io/projected/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-kube-api-access-8m5sx\") pod \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") "
Oct 30 20:00:23.520654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.520577    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-maas-api-service-ca\") pod \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\" (UID: \"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59\") "
Oct 30 20:00:23.520791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.520770    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a278a416-54cd-4256-ac17-6e23b31c0671-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a278a416-54cd-4256-ac17-6e23b31c0671-maas-api-service-ca\") pod \"a278a416-54cd-4256-ac17-6e23b31c0671\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") "
Oct 30 20:00:23.520866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.520837    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a278a416-54cd-4256-ac17-6e23b31c0671-kube-api-access-7cc26\" (UniqueName: \"kubernetes.io/projected/a278a416-54cd-4256-ac17-6e23b31c0671-kube-api-access-7cc26\") pod \"a278a416-54cd-4256-ac17-6e23b31c0671\" (UID: \"a278a416-54cd-4256-ac17-6e23b31c0671\") "
Oct 30 20:00:23.521910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.521884    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7g4\" (UniqueName: \"kubernetes.io/projected/32cb5bde-e08d-496d-a83b-365d19d8d4a4-kube-api-access-sk7g4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.523113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.523082    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a278a416-54cd-4256-ac17-6e23b31c0671-maas-api-service-ca" pod "a278a416-54cd-4256-ac17-6e23b31c0671" (UID: "a278a416-54cd-4256-ac17-6e23b31c0671"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.523678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.523652    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-maas-api-service-ca" pod "2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" (UID: "2ea9d0ec-fb50-4c79-b2d0-1ec024506b59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.525402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.525377    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-kube-api-access-8m5sx" pod "2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" (UID: "2ea9d0ec-fb50-4c79-b2d0-1ec024506b59"). InnerVolumeSpecName "kube-api-access-8m5sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.529337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.529296    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a278a416-54cd-4256-ac17-6e23b31c0671-kube-api-access-7cc26" pod "a278a416-54cd-4256-ac17-6e23b31c0671" (UID: "a278a416-54cd-4256-ac17-6e23b31c0671"). InnerVolumeSpecName "kube-api-access-7cc26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.616407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.616366    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" path="/var/lib/kubelet/pods/032754c4-d986-4767-bc09-19ff1ba923a7/volumes"
Oct 30 20:00:23.617151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.617117    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" path="/var/lib/kubelet/pods/64d4d50d-febd-43fe-90dc-cb8836687123/volumes"
Oct 30 20:00:23.617687 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.617667    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906f4700-7e26-4304-be69-bd6c77566780" path="/var/lib/kubelet/pods/906f4700-7e26-4304-be69-bd6c77566780/volumes"
Oct 30 20:00:23.618434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.618418    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" path="/var/lib/kubelet/pods/c60757fe-86c3-4d66-a25c-e1ce2a276453/volumes"
Oct 30 20:00:23.618875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.618861    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" path="/var/lib/kubelet/pods/d7d4bc1d-85a2-4553-942d-dbc161fde6b7/volumes"
Oct 30 20:00:23.619311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.619296    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" path="/var/lib/kubelet/pods/e6456596-13e7-48e2-af8b-1ec739ca9871/volumes"
Oct 30 20:00:23.625528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.625447    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a278a416-54cd-4256-ac17-6e23b31c0671-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.625528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.625507    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cc26\" (UniqueName: \"kubernetes.io/projected/a278a416-54cd-4256-ac17-6e23b31c0671-kube-api-access-7cc26\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.625528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.625524    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5sx\" (UniqueName: \"kubernetes.io/projected/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-kube-api-access-8m5sx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.625745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.625539    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.659197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.659163    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:23.726255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.726224    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-maas-api-service-ca\") pod \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") "
Oct 30 20:00:23.726423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.726373    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-kube-api-access-5szkt\" (UniqueName: \"kubernetes.io/projected/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-kube-api-access-5szkt\") pod \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\" (UID: \"badf3d12-c146-4ab3-b2b7-3c2fc1af5574\") "
Oct 30 20:00:23.726708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.726678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-maas-api-service-ca" pod "badf3d12-c146-4ab3-b2b7-3c2fc1af5574" (UID: "badf3d12-c146-4ab3-b2b7-3c2fc1af5574"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.727173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.727147    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.728811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.728789    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-kube-api-access-5szkt" pod "badf3d12-c146-4ab3-b2b7-3c2fc1af5574" (UID: "badf3d12-c146-4ab3-b2b7-3c2fc1af5574"). InnerVolumeSpecName "kube-api-access-5szkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.829162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:23.829128    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5szkt\" (UniqueName: \"kubernetes.io/projected/badf3d12-c146-4ab3-b2b7-3c2fc1af5574-kube-api-access-5szkt\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:24.139090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.139058    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"
Oct 30 20:00:24.139777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.139071    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2" event={"ID":"badf3d12-c146-4ab3-b2b7-3c2fc1af5574","Type":"ContainerDied","Data":"3c212e562ca82b72c8cc68fe098813fc80bb6e11be6d294b56e8281ff3d52b04"}
Oct 30 20:00:24.139888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.139835    4519 scope.go:122] "RemoveContainer" containerID="a3cfb109407e4c0cd8de13189da3803105768d22e1675f4e3118fffd1789ab02"
Oct 30 20:00:24.143061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.143031    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5" event={"ID":"a278a416-54cd-4256-ac17-6e23b31c0671","Type":"ContainerDied","Data":"4e25365d502ec47fdd9cee3c66e2c40e3589a24f38a5b97adea30d048ac10700"}
Oct 30 20:00:24.143197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.143166    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"
Oct 30 20:00:24.147692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.147651    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"
Oct 30 20:00:24.148023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.147979    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k" event={"ID":"2ea9d0ec-fb50-4c79-b2d0-1ec024506b59","Type":"ContainerDied","Data":"7d107e54509d80bf593dba22a83c8c74ee1dda471b78cc6ab3425637ef4bc4cb"}
Oct 30 20:00:24.155561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.155532    4519 scope.go:122] "RemoveContainer" containerID="da93cfda902a96da7e6dfdfccd9d8444197cbb557bee8dfa5613b8528e524184"
Oct 30 20:00:24.156655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.156623    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc" event={"ID":"32cb5bde-e08d-496d-a83b-365d19d8d4a4","Type":"ContainerDied","Data":"942e8b247081ae01233e0c2b31cbf5683826742ddb6a36df48ec5022beecb069"}
Oct 30 20:00:24.156780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.156744    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"
Oct 30 20:00:24.167798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.167768    4519 scope.go:122] "RemoveContainer" containerID="42bbe6c830c5bd51e0193f1962af49158de9d4b9977d0ff970438105783bb5a7"
Oct 30 20:00:24.179391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.179361    4519 scope.go:122] "RemoveContainer" containerID="4bd39529858b48392b3b1919fce19381d9dd3921ca036186d250425d691efb4e"
Oct 30 20:00:24.194456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.194426    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"]
Oct 30 20:00:24.200018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.199990    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8d97a5-29889840-px44k"]
Oct 30 20:00:24.258060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.258018    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"]
Oct 30 20:00:24.263374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.263342    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889840-mgzg5"]
Oct 30 20:00:24.290462 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.290427    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"]
Oct 30 20:00:24.297986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.297943    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-bbbl2"]
Oct 30 20:00:24.335623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.335563    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"]
Oct 30 20:00:24.337323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:24.337293    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889840-hnrhc"]
Oct 30 20:00:25.523996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.523955    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"]
Oct 30 20:00:25.524960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.524935    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.524962    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.524978    4519 cpu_manager.go:414] "containerMap: removing container" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.524986    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525001    4519 cpu_manager.go:414] "containerMap: removing container" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525010    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525022    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525029    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525042    4519 cpu_manager.go:414] "containerMap: removing container" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525049    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.525060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525058    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525065    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525077    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525084    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525101    4519 cpu_manager.go:414] "containerMap: removing container" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525108    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525118    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525125    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525133    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525140    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525151    4519 cpu_manager.go:414] "containerMap: removing container" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525157    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525169    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525176    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525191    4519 cpu_manager.go:414] "containerMap: removing container" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525197    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525210    4519 cpu_manager.go:414] "containerMap: removing container" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525217    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525227    4519 cpu_manager.go:414] "containerMap: removing container" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525234    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525246    4519 cpu_manager.go:414] "containerMap: removing container" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525254    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525264    4519 cpu_manager.go:414] "containerMap: removing container" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525271    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525293    4519 cpu_manager.go:414] "containerMap: removing container" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525300    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525310    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525319    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525330    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525338    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525367    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525375    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525385    4519 cpu_manager.go:414] "containerMap: removing container" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525392    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525403    4519 cpu_manager.go:414] "containerMap: removing container" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525410    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525420    4519 cpu_manager.go:414] "containerMap: removing container" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:25.525420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525428    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525445    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525452    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525479    4519 cpu_manager.go:414] "containerMap: removing container" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525486    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525497    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525503    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525514    4519 cpu_manager.go:414] "containerMap: removing container" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525520    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525537    4519 cpu_manager.go:414] "containerMap: removing container" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525543    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525557    4519 cpu_manager.go:414] "containerMap: removing container" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525564    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525573    4519 cpu_manager.go:414] "containerMap: removing container" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525579    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525600    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525607    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525619    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525625    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525635    4519 cpu_manager.go:414] "containerMap: removing container" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525642    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525657    4519 cpu_manager.go:414] "containerMap: removing container" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525663    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525676    4519 cpu_manager.go:414] "containerMap: removing container" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525682    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525846    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525859    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525868    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525878    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525887    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525894    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525905    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525914    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525923    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525932    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525940    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525950    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525959    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525967    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525976    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525983    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525990    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.525999    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526007    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526016    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526025    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526035    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526047    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526055    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526065    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526073    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526083    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526091    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526101    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526110    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526119    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526129    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526139    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ec8783-2c39-4c35-836a-89a002783adb" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526149    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526158    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:25.526896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.526168    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:25.532383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.532357    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.594149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.593415    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:00:25.594149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.594096    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:25.841013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.801817    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxxf\" (UniqueName: \"kubernetes.io/projected/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-kube-api-access-zzxxf\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.841013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.801999    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.841013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.802057    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.847258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.847221    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" path="/var/lib/kubelet/pods/2ea9d0ec-fb50-4c79-b2d0-1ec024506b59/volumes"
Oct 30 20:00:25.848285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.848258    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" path="/var/lib/kubelet/pods/32cb5bde-e08d-496d-a83b-365d19d8d4a4/volumes"
Oct 30 20:00:25.879488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.874895    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" path="/var/lib/kubelet/pods/a278a416-54cd-4256-ac17-6e23b31c0671/volumes"
Oct 30 20:00:25.879488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.875683    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" path="/var/lib/kubelet/pods/badf3d12-c146-4ab3-b2b7-3c2fc1af5574/volumes"
Oct 30 20:00:25.879488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.876196    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"]
Oct 30 20:00:25.904490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.904179    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.904490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.904268    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.904730 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.904593    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxxf\" (UniqueName: \"kubernetes.io/projected/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-kube-api-access-zzxxf\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.907963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.906095    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.908992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.908597    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:25.925675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:25.925641    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxxf\" (UniqueName: \"kubernetes.io/projected/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-kube-api-access-zzxxf\") pod \"maas-api-revoke-sub-39f00559-sdv4s\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:26.172803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.172767    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:26.573763 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:26.573726    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec21be5_5c47_4949_82fb_3d2322ac0ec7.slice/crio-b611ea6bb34886c90076f1baa845a0599f553b4682d0f28a637cc2a7d9a6806b WatchSource:0}: Error finding container b611ea6bb34886c90076f1baa845a0599f553b4682d0f28a637cc2a7d9a6806b: Status 404 returned error can't find the container with id b611ea6bb34886c90076f1baa845a0599f553b4682d0f28a637cc2a7d9a6806b
Oct 30 20:00:26.576145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.575860    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"]
Oct 30 20:00:26.708206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.708165    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"]
Oct 30 20:00:26.708983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.708961    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:26.708983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.708984    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.708997    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709003    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709013    4519 cpu_manager.go:414] "containerMap: removing container" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709017    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32248e1e-4b52-4f73-b987-b0e348fab854" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709030    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709035    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709043    4519 cpu_manager.go:414] "containerMap: removing container" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709048    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cb5bde-e08d-496d-a83b-365d19d8d4a4" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709065    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709072    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709083    4519 cpu_manager.go:414] "containerMap: removing container" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709088    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709099    4519 cpu_manager.go:414] "containerMap: removing container" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709103    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709113    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:26.709112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709117    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709125    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709130    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a278a416-54cd-4256-ac17-6e23b31c0671" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709137    4519 cpu_manager.go:414] "containerMap: removing container" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709141    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709148    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709152    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709243    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d4d50d-febd-43fe-90dc-cb8836687123" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709252    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="032754c4-d986-4767-bc09-19ff1ba923a7" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709258    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d4bc1d-85a2-4553-942d-dbc161fde6b7" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709267    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf3d12-c146-4ab3-b2b7-3c2fc1af5574" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709275    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b31c7a-ff39-4203-9e69-e8b773b009e4" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709282    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef72380-91f7-4631-8100-4454165123fc" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709292    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60757fe-86c3-4d66-a25c-e1ce2a276453" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709302    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dc526f-7dee-49ac-b389-b340ceab3266" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709311    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2324b1-81c5-46d8-adec-73e2aebbe111" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709320    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="906f4700-7e26-4304-be69-bd6c77566780" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709326    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6456596-13e7-48e2-af8b-1ec739ca9871" containerName="cleanup"
Oct 30 20:00:26.709611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.709334    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea9d0ec-fb50-4c79-b2d0-1ec024506b59" containerName="cleanup"
Oct 30 20:00:26.714319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.714290    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.720170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.720138    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lxr\" (UniqueName: \"kubernetes.io/projected/2c104561-b3ad-4197-b6fd-4227e414f091-kube-api-access-p9lxr\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.720336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.720312    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.720536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.720512    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.732896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.732860    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"]
Oct 30 20:00:26.822559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.822425    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lxr\" (UniqueName: \"kubernetes.io/projected/2c104561-b3ad-4197-b6fd-4227e414f091-kube-api-access-p9lxr\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.822810 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.822614    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.822810 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.822736    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.823284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.823264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.825285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.825253    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:26.843143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:26.843096    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lxr\" (UniqueName: \"kubernetes.io/projected/2c104561-b3ad-4197-b6fd-4227e414f091-kube-api-access-p9lxr\") pod \"maas-api-revoke-sub-40400581-cs58v\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:27.032209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:27.032172    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:27.208422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:27.208379    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s" event={"ID":"1ec21be5-5c47-4949-82fb-3d2322ac0ec7","Type":"ContainerStarted","Data":"e7d6c908d04381609a257d198240d34eac55ae164f8a24515a75f281954242dd"}
Oct 30 20:00:27.208648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:27.208450    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s" event={"ID":"1ec21be5-5c47-4949-82fb-3d2322ac0ec7","Type":"ContainerStarted","Data":"b611ea6bb34886c90076f1baa845a0599f553b4682d0f28a637cc2a7d9a6806b"}
Oct 30 20:00:27.253230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:27.253169    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s" podStartSLOduration=2.253149755 podStartE2EDuration="2.253149755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:27.248620717 +0000 UTC m=+1551.749324452" watchObservedRunningTime="2026-10-30 20:00:27.253149755 +0000 UTC m=+1551.753853478"
Oct 30 20:00:27.483130 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:27.483094    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c104561_b3ad_4197_b6fd_4227e414f091.slice/crio-bae7fcb450bc36920ed347564a81700e6e449e6f46adf422b0604fbe51d7dca8 WatchSource:0}: Error finding container bae7fcb450bc36920ed347564a81700e6e449e6f46adf422b0604fbe51d7dca8: Status 404 returned error can't find the container with id bae7fcb450bc36920ed347564a81700e6e449e6f46adf422b0604fbe51d7dca8
Oct 30 20:00:27.483998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:27.483951    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"]
Oct 30 20:00:28.220617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:28.220582    4519 generic.go:356] "Generic (PLEG): container finished" podID="1ec21be5-5c47-4949-82fb-3d2322ac0ec7" containerID="e7d6c908d04381609a257d198240d34eac55ae164f8a24515a75f281954242dd" exitCode=0
Oct 30 20:00:28.221099 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:28.220663    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s" event={"ID":"1ec21be5-5c47-4949-82fb-3d2322ac0ec7","Type":"ContainerDied","Data":"e7d6c908d04381609a257d198240d34eac55ae164f8a24515a75f281954242dd"}
Oct 30 20:00:28.222508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:28.222480    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v" event={"ID":"2c104561-b3ad-4197-b6fd-4227e414f091","Type":"ContainerStarted","Data":"b8846c446639c744c3c5f5b19e9abac64acdd66556549e033919ce72b7cbf49f"}
Oct 30 20:00:28.222633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:28.222532    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v" event={"ID":"2c104561-b3ad-4197-b6fd-4227e414f091","Type":"ContainerStarted","Data":"bae7fcb450bc36920ed347564a81700e6e449e6f46adf422b0604fbe51d7dca8"}
Oct 30 20:00:28.303164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:28.303107    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v" podStartSLOduration=2.303090511 podStartE2EDuration="2.303090511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:28.298773672 +0000 UTC m=+1552.799477397" watchObservedRunningTime="2026-10-30 20:00:28.303090511 +0000 UTC m=+1552.803794235"
Oct 30 20:00:29.443489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.415287    4519 generic.go:356] "Generic (PLEG): container finished" podID="2c104561-b3ad-4197-b6fd-4227e414f091" containerID="b8846c446639c744c3c5f5b19e9abac64acdd66556549e033919ce72b7cbf49f" exitCode=0
Oct 30 20:00:29.443489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.416384    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v" event={"ID":"2c104561-b3ad-4197-b6fd-4227e414f091","Type":"ContainerDied","Data":"b8846c446639c744c3c5f5b19e9abac64acdd66556549e033919ce72b7cbf49f"}
Oct 30 20:00:29.666840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.666801    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:29.817044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.816952    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-kube-api-access-zzxxf\" (UniqueName: \"kubernetes.io/projected/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-kube-api-access-zzxxf\") pod \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") "
Oct 30 20:00:29.817251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.817094    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-xks-ca\") pod \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") "
Oct 30 20:00:29.817251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.817196    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-openshift-ca\") pod \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\" (UID: \"1ec21be5-5c47-4949-82fb-3d2322ac0ec7\") "
Oct 30 20:00:29.817643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.817603    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-openshift-ca" pod "1ec21be5-5c47-4949-82fb-3d2322ac0ec7" (UID: "1ec21be5-5c47-4949-82fb-3d2322ac0ec7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:29.818175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.818141    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:29.819441 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.819412    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-xks-ca" pod "1ec21be5-5c47-4949-82fb-3d2322ac0ec7" (UID: "1ec21be5-5c47-4949-82fb-3d2322ac0ec7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:29.819557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.819445    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-kube-api-access-zzxxf" pod "1ec21be5-5c47-4949-82fb-3d2322ac0ec7" (UID: "1ec21be5-5c47-4949-82fb-3d2322ac0ec7"). InnerVolumeSpecName "kube-api-access-zzxxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:29.920383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.920339    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:29.920383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:29.920381    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxxf\" (UniqueName: \"kubernetes.io/projected/1ec21be5-5c47-4949-82fb-3d2322ac0ec7-kube-api-access-zzxxf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.425543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.425516    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"
Oct 30 20:00:30.425543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.425522    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s" event={"ID":"1ec21be5-5c47-4949-82fb-3d2322ac0ec7","Type":"ContainerDied","Data":"b611ea6bb34886c90076f1baa845a0599f553b4682d0f28a637cc2a7d9a6806b"}
Oct 30 20:00:30.425790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.425590    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b611ea6bb34886c90076f1baa845a0599f553b4682d0f28a637cc2a7d9a6806b"
Oct 30 20:00:30.714457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.714428    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:30.842873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.842837    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-xks-ca\") pod \"2c104561-b3ad-4197-b6fd-4227e414f091\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") "
Oct 30 20:00:30.843061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.842895    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-openshift-ca\") pod \"2c104561-b3ad-4197-b6fd-4227e414f091\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") "
Oct 30 20:00:30.843061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.842927    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c104561-b3ad-4197-b6fd-4227e414f091-kube-api-access-p9lxr\" (UniqueName: \"kubernetes.io/projected/2c104561-b3ad-4197-b6fd-4227e414f091-kube-api-access-p9lxr\") pod \"2c104561-b3ad-4197-b6fd-4227e414f091\" (UID: \"2c104561-b3ad-4197-b6fd-4227e414f091\") "
Oct 30 20:00:30.843368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.843330    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-openshift-ca" pod "2c104561-b3ad-4197-b6fd-4227e414f091" (UID: "2c104561-b3ad-4197-b6fd-4227e414f091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:30.843763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.843744    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.845593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.845569    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-xks-ca" pod "2c104561-b3ad-4197-b6fd-4227e414f091" (UID: "2c104561-b3ad-4197-b6fd-4227e414f091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:30.846058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.846042    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c104561-b3ad-4197-b6fd-4227e414f091-kube-api-access-p9lxr" pod "2c104561-b3ad-4197-b6fd-4227e414f091" (UID: "2c104561-b3ad-4197-b6fd-4227e414f091"). InnerVolumeSpecName "kube-api-access-p9lxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:30.945020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.944925    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c104561-b3ad-4197-b6fd-4227e414f091-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.945020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:30.944958    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lxr\" (UniqueName: \"kubernetes.io/projected/2c104561-b3ad-4197-b6fd-4227e414f091-kube-api-access-p9lxr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:31.007494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.007446    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"]
Oct 30 20:00:31.008414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.008392    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2c104561-b3ad-4197-b6fd-4227e414f091" containerName="revoke-keys"
Oct 30 20:00:31.008500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.008417    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c104561-b3ad-4197-b6fd-4227e414f091" containerName="revoke-keys"
Oct 30 20:00:31.008500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.008456    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1ec21be5-5c47-4949-82fb-3d2322ac0ec7" containerName="revoke-keys"
Oct 30 20:00:31.008607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.008540    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec21be5-5c47-4949-82fb-3d2322ac0ec7" containerName="revoke-keys"
Oct 30 20:00:31.008788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.008698    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c104561-b3ad-4197-b6fd-4227e414f091" containerName="revoke-keys"
Oct 30 20:00:31.008788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.008720    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec21be5-5c47-4949-82fb-3d2322ac0ec7" containerName="revoke-keys"
Oct 30 20:00:31.012788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.012766    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.030279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.029301    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ef132f-openshift-default-dockercfg-474gr\""
Oct 30 20:00:31.071573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.071535    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"]
Oct 30 20:00:31.149305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149267    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-certs\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149344    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-data\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149446    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96md\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-kube-api-access-h96md\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149487    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-podinfo\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149562    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-token\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149709    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-envoy\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149745    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-credential-socket\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149915    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-socket\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.149994 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.149947    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istiod-ca-cert\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.258754    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-envoy\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.258821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-credential-socket\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.258987    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-socket\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.259032    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istiod-ca-cert\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.259109    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-certs\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.259175    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-data\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.259274    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96md\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-kube-api-access-h96md\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.259313    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-podinfo\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.261590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.259426    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-token\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.266507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.263379    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istiod-ca-cert\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.266507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.263984    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-credential-socket\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.266507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.264226    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-socket\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.266507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.264446    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-data\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.266507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.264698    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-certs\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.309549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.301450    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-podinfo\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.310913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.310882    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-envoy\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.334518 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.322789    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96md\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-kube-api-access-h96md\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.350506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.346453    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-token\") pod \"e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") " pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.396632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.380784    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:31.524552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.504210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v" event={"ID":"2c104561-b3ad-4197-b6fd-4227e414f091","Type":"ContainerDied","Data":"bae7fcb450bc36920ed347564a81700e6e449e6f46adf422b0604fbe51d7dca8"}
Oct 30 20:00:31.524552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.504281    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bae7fcb450bc36920ed347564a81700e6e449e6f46adf422b0604fbe51d7dca8"
Oct 30 20:00:31.524552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.504354    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"
Oct 30 20:00:31.974040 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:31.973995    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8b2862_9442_47d5_af3c_ff55bb26b17c.slice/crio-385b9a4c45ee48745e7bc5d8376fa485ced31ea2489c858ea03f21f11372d0b9 WatchSource:0}: Error finding container 385b9a4c45ee48745e7bc5d8376fa485ced31ea2489c858ea03f21f11372d0b9: Status 404 returned error can't find the container with id 385b9a4c45ee48745e7bc5d8376fa485ced31ea2489c858ea03f21f11372d0b9
Oct 30 20:00:31.974433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.974131    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"]
Oct 30 20:00:31.977930 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.977896    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:31.978051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.977984    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:31.978051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:31.978018    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:32.295108 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:32.295017    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:00:32.514759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:32.514720    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" event={"ID":"aa8b2862-9442-47d5-af3c-ff55bb26b17c","Type":"ContainerStarted","Data":"0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c"}
Oct 30 20:00:32.514939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:32.514803    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" event={"ID":"aa8b2862-9442-47d5-af3c-ff55bb26b17c","Type":"ContainerStarted","Data":"385b9a4c45ee48745e7bc5d8376fa485ced31ea2489c858ea03f21f11372d0b9"}
Oct 30 20:00:32.561134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:32.561001    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podStartSLOduration=2.560983599 podStartE2EDuration="2.560983599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:32.559478749 +0000 UTC m=+1557.060182469" watchObservedRunningTime="2026-10-30 20:00:32.560983599 +0000 UTC m=+1557.061687325"
Oct 30 20:00:33.382615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:33.382561    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:33.390458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:33.390423    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:33.525225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:33.525183    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:33.526373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:33.526345    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:00:41.615438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:41.615402    4519 scope.go:122] "RemoveContainer" containerID="b5f56586f066cc19fe54b617c8d2f740dff56da56495597ed1d67621dd61834d"
Oct 30 20:00:41.627548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:41.627522    4519 scope.go:122] "RemoveContainer" containerID="b137dfd03ab4aebdeae57ba17d8ad1f96aff974fb063cbf2c7a166e8fbe033c6"
Oct 30 20:00:41.638447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:41.638422    4519 scope.go:122] "RemoveContainer" containerID="0498430f27f0ce8727e1735032b2eae2143c1d721aeafd155f7baf6068736d2a"
Oct 30 20:00:41.649551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:41.649522    4519 scope.go:122] "RemoveContainer" containerID="40e2a0d058a5f55aa903a5b280dbd2b32339804cadbed13bf9143ecdb8a98769"
Oct 30 20:00:43.714776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:43.714736    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"]
Oct 30 20:00:43.729669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:43.729629    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d97a5-fvrtr"]
Oct 30 20:00:45.676657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:45.676613    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1320673f-eaec-48e3-8bb1-3ad0c9e2a94e" path="/var/lib/kubelet/pods/1320673f-eaec-48e3-8bb1-3ad0c9e2a94e/volumes"
Oct 30 20:00:48.029374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:48.029333    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"]
Oct 30 20:00:48.034079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:48.034052    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-93263c-558cb"]
Oct 30 20:00:49.616282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:49.616200    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355594ec-cf9f-4148-bca2-4313dfc25b89" path="/var/lib/kubelet/pods/355594ec-cf9f-4148-bca2-4313dfc25b89/volumes"
Oct 30 20:00:51.426628 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:51.426582    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"]
Oct 30 20:00:51.427182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:51.426913    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy" containerID="cri-o://0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb" gracePeriod=30
Oct 30 20:00:52.005774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.005736    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"]
Oct 30 20:00:52.012701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.012678    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.029682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.029648    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"]
Oct 30 20:00:52.103787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.103757    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:00:52.103787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.103833    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:52.116981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.116944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.117197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.117063    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hxp\" (UniqueName: \"kubernetes.io/projected/e40857ad-a571-4697-95ec-4c3be56da5c1-kube-api-access-c6hxp\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.117263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.117252    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.218593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.218549    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hxp\" (UniqueName: \"kubernetes.io/projected/e40857ad-a571-4697-95ec-4c3be56da5c1-kube-api-access-c6hxp\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.218784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.218737    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.218879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.218863    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.219551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.219521    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.221321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.221298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.238176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.238140    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hxp\" (UniqueName: \"kubernetes.io/projected/e40857ad-a571-4697-95ec-4c3be56da5c1-kube-api-access-c6hxp\") pod \"maas-api-revoke-keys-e2e-api-b-d96913-rwpb8\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.329296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.329217    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:52.639338 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:52.639293    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40857ad_a571_4697_95ec_4c3be56da5c1.slice/crio-1d9433cd8871d8b5ab80fa2a5c0c700d25f3c0cb37e97c08948da32c1b90d3c0 WatchSource:0}: Error finding container 1d9433cd8871d8b5ab80fa2a5c0c700d25f3c0cb37e97c08948da32c1b90d3c0: Status 404 returned error can't find the container with id 1d9433cd8871d8b5ab80fa2a5c0c700d25f3c0cb37e97c08948da32c1b90d3c0
Oct 30 20:00:52.641815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.641780    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"]
Oct 30 20:00:52.838936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:52.838892    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8" event={"ID":"e40857ad-a571-4697-95ec-4c3be56da5c1","Type":"ContainerStarted","Data":"1d9433cd8871d8b5ab80fa2a5c0c700d25f3c0cb37e97c08948da32c1b90d3c0"}
Oct 30 20:00:53.849250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:53.849149    4519 generic.go:356] "Generic (PLEG): container finished" podID="e40857ad-a571-4697-95ec-4c3be56da5c1" containerID="5e765b318da18d9ab8564cc12414a6364ee6c083efa2187cd7a36186d8faaaa0" exitCode=0
Oct 30 20:00:53.849735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:53.849259    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8" event={"ID":"e40857ad-a571-4697-95ec-4c3be56da5c1","Type":"ContainerDied","Data":"5e765b318da18d9ab8564cc12414a6364ee6c083efa2187cd7a36186d8faaaa0"}
Oct 30 20:00:55.391924 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.391895    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:55.511675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.511638    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-xks-ca\") pod \"e40857ad-a571-4697-95ec-4c3be56da5c1\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") "
Oct 30 20:00:55.511877 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.511792    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-openshift-ca\") pod \"e40857ad-a571-4697-95ec-4c3be56da5c1\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") "
Oct 30 20:00:55.512015 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.512000    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40857ad-a571-4697-95ec-4c3be56da5c1-kube-api-access-c6hxp\" (UniqueName: \"kubernetes.io/projected/e40857ad-a571-4697-95ec-4c3be56da5c1-kube-api-access-c6hxp\") pod \"e40857ad-a571-4697-95ec-4c3be56da5c1\" (UID: \"e40857ad-a571-4697-95ec-4c3be56da5c1\") "
Oct 30 20:00:55.512196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.512171    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-openshift-ca" pod "e40857ad-a571-4697-95ec-4c3be56da5c1" (UID: "e40857ad-a571-4697-95ec-4c3be56da5c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:55.512757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.512733    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:55.514006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.513979    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-xks-ca" pod "e40857ad-a571-4697-95ec-4c3be56da5c1" (UID: "e40857ad-a571-4697-95ec-4c3be56da5c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:55.514141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.514123    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40857ad-a571-4697-95ec-4c3be56da5c1-kube-api-access-c6hxp" pod "e40857ad-a571-4697-95ec-4c3be56da5c1" (UID: "e40857ad-a571-4697-95ec-4c3be56da5c1"). InnerVolumeSpecName "kube-api-access-c6hxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:55.614729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.614632    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40857ad-a571-4697-95ec-4c3be56da5c1-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:55.614729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.614669    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hxp\" (UniqueName: \"kubernetes.io/projected/e40857ad-a571-4697-95ec-4c3be56da5c1-kube-api-access-c6hxp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:55.871284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.871174    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8" event={"ID":"e40857ad-a571-4697-95ec-4c3be56da5c1","Type":"ContainerDied","Data":"1d9433cd8871d8b5ab80fa2a5c0c700d25f3c0cb37e97c08948da32c1b90d3c0"}
Oct 30 20:00:55.871284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.871252    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d9433cd8871d8b5ab80fa2a5c0c700d25f3c0cb37e97c08948da32c1b90d3c0"
Oct 30 20:00:55.871536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.871322    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"
Oct 30 20:00:55.972864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.972826    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"]
Oct 30 20:00:55.973435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:55.973401    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy" containerID="cri-o://c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14" gracePeriod=30
Oct 30 20:00:56.612275 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.612236    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"]
Oct 30 20:00:56.613926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.613892    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e40857ad-a571-4697-95ec-4c3be56da5c1" containerName="revoke-keys"
Oct 30 20:00:56.614566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.613927    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40857ad-a571-4697-95ec-4c3be56da5c1" containerName="revoke-keys"
Oct 30 20:00:56.614566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.614140    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40857ad-a571-4697-95ec-4c3be56da5c1" containerName="revoke-keys"
Oct 30 20:00:56.651626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.650727    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.652167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.652080    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"]
Oct 30 20:00:56.665666 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.665635    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:56.666627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.666019    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:00:56.751147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.751104    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.751311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.751206    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.751574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.751546    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchkq\" (UniqueName: \"kubernetes.io/projected/471dfd1b-61fa-43ab-b337-415594a0b4f1-kube-api-access-hchkq\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.754218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.754196    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 20:00:56.854096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854054    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-credential-socket\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854166    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-envoy\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854213    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fc2d597-f684-4579-ba47-7735c112706a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc2d597-f684-4579-ba47-7735c112706a-istio-podinfo\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854318    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-kube-api-access-5tbqf\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-kube-api-access-5tbqf\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854379    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-socket\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854430    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-certs\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854518    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc2d597-f684-4579-ba47-7735c112706a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc2d597-f684-4579-ba47-7735c112706a-istiod-ca-cert\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854631    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-data\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854686    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-istio-token\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-istio-token\") pod \"2fc2d597-f684-4579-ba47-7735c112706a\" (UID: \"2fc2d597-f684-4579-ba47-7735c112706a\") "
Oct 30 20:00:56.854927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.854907    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchkq\" (UniqueName: \"kubernetes.io/projected/471dfd1b-61fa-43ab-b337-415594a0b4f1-kube-api-access-hchkq\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.855189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.855175    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.855232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.855220    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.856536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.856498    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-credential-socket" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:56.856746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.856722    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc2d597-f684-4579-ba47-7735c112706a-istiod-ca-cert" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:56.857085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.857026    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-data" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:56.858028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.857600    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-socket" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:56.858184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.858160    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-certs" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:56.858573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.858548    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.859322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.859297    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fc2d597-f684-4579-ba47-7735c112706a-istio-podinfo" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:56.859650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.859623    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-envoy" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:56.860752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.860697    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-istio-token" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:56.860934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.860909    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-kube-api-access-5tbqf" pod "2fc2d597-f684-4579-ba47-7735c112706a" (UID: "2fc2d597-f684-4579-ba47-7735c112706a"). InnerVolumeSpecName "kube-api-access-5tbqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:56.862949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.862921    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.881502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.878630    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchkq\" (UniqueName: \"kubernetes.io/projected/471dfd1b-61fa-43ab-b337-415594a0b4f1-kube-api-access-hchkq\") pod \"maas-api-revoke-keys-e2e-api-b-540a43-5b4nq\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:56.886020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.885984    4519 generic.go:356] "Generic (PLEG): container finished" podID="2fc2d597-f684-4579-ba47-7735c112706a" containerID="0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb" exitCode=0
Oct 30 20:00:56.886248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.886229    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" event={"ID":"2fc2d597-f684-4579-ba47-7735c112706a","Type":"ContainerDied","Data":"0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb"}
Oct 30 20:00:56.886305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.886285    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb" event={"ID":"2fc2d597-f684-4579-ba47-7735c112706a","Type":"ContainerDied","Data":"0482dba3fdd1ce981a9e784479cc952bf80e640589c56985a30c6235ada75bf6"}
Oct 30 20:00:56.886351 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.886315    4519 scope.go:122] "RemoveContainer" containerID="0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb"
Oct 30 20:00:56.886565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.886551    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"
Oct 30 20:00:56.907812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.907783    4519 scope.go:122] "RemoveContainer" containerID="76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977"
Oct 30 20:00:56.925972 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.925946    4519 scope.go:122] "RemoveContainer" containerID="0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb"
Oct 30 20:00:56.926313 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:56.926287    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb\": container with ID starting with 0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb not found: ID does not exist" containerID="0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb"
Oct 30 20:00:56.926410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.926322    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb"} err="failed to get container status \"0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb\": rpc error: code = NotFound desc = could not find container \"0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb\": container with ID starting with 0ed14c0a996815b97f9d95519bdfa2692999aa524cee24330893df447e8742eb not found: ID does not exist"
Oct 30 20:00:56.926410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.926345    4519 scope.go:122] "RemoveContainer" containerID="76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977"
Oct 30 20:00:56.926671 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:00:56.926647    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977\": container with ID starting with 76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977 not found: ID does not exist" containerID="76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977"
Oct 30 20:00:56.926716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.926684    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977"} err="failed to get container status \"76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977\": rpc error: code = NotFound desc = could not find container \"76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977\": container with ID starting with 76af6510af0cba2aba4bb2e1787998a245d0fab40a28b7244afc2baf86700977 not found: ID does not exist"
Oct 30 20:00:56.954809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.954747    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"]
Oct 30 20:00:56.957951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.957917    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.957951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.957952    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.958193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.957968    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.958193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.957982    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.958193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.957996    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc2d597-f684-4579-ba47-7735c112706a-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.958193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.958012    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbqf\" (UniqueName: \"kubernetes.io/projected/2fc2d597-f684-4579-ba47-7735c112706a-kube-api-access-5tbqf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.958193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.958024    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.958193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.958036    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc2d597-f684-4579-ba47-7735c112706a-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.958193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.958050    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc2d597-f684-4579-ba47-7735c112706a-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:00:56.963484 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.962876    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8d97a5-openshift-default-75d7b87765-s9pbb"]
Oct 30 20:00:56.977572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:56.970502    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:00:57.377962 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:00:57.376924    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471dfd1b_61fa_43ab_b337_415594a0b4f1.slice/crio-6beb73d57a427ebf0503abafaa4c8afd5212bd26377c13a9f57623c13ad387ee WatchSource:0}: Error finding container 6beb73d57a427ebf0503abafaa4c8afd5212bd26377c13a9f57623c13ad387ee: Status 404 returned error can't find the container with id 6beb73d57a427ebf0503abafaa4c8afd5212bd26377c13a9f57623c13ad387ee
Oct 30 20:00:57.378190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:57.378087    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"]
Oct 30 20:00:57.616088 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:57.616056    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc2d597-f684-4579-ba47-7735c112706a" path="/var/lib/kubelet/pods/2fc2d597-f684-4579-ba47-7735c112706a/volumes"
Oct 30 20:00:57.972092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:57.972002    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq" event={"ID":"471dfd1b-61fa-43ab-b337-415594a0b4f1","Type":"ContainerStarted","Data":"6beb73d57a427ebf0503abafaa4c8afd5212bd26377c13a9f57623c13ad387ee"}
Oct 30 20:00:59.004241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:59.004196    4519 generic.go:356] "Generic (PLEG): container finished" podID="471dfd1b-61fa-43ab-b337-415594a0b4f1" containerID="1aad72e0de40626fa19233d3d3cea026c9469a5088ecbd69f5bfac4975501dc3" exitCode=0
Oct 30 20:00:59.004775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:00:59.004405    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq" event={"ID":"471dfd1b-61fa-43ab-b337-415594a0b4f1","Type":"ContainerDied","Data":"1aad72e0de40626fa19233d3d3cea026c9469a5088ecbd69f5bfac4975501dc3"}
Oct 30 20:01:00.227261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.227229    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:01:00.303667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.303565    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471dfd1b-61fa-43ab-b337-415594a0b4f1-kube-api-access-hchkq\" (UniqueName: \"kubernetes.io/projected/471dfd1b-61fa-43ab-b337-415594a0b4f1-kube-api-access-hchkq\") pod \"471dfd1b-61fa-43ab-b337-415594a0b4f1\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") "
Oct 30 20:01:00.303667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.303641    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-openshift-ca\") pod \"471dfd1b-61fa-43ab-b337-415594a0b4f1\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") "
Oct 30 20:01:00.303911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.303719    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-xks-ca\") pod \"471dfd1b-61fa-43ab-b337-415594a0b4f1\" (UID: \"471dfd1b-61fa-43ab-b337-415594a0b4f1\") "
Oct 30 20:01:00.305092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.305059    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-openshift-ca" pod "471dfd1b-61fa-43ab-b337-415594a0b4f1" (UID: "471dfd1b-61fa-43ab-b337-415594a0b4f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:00.307025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.307000    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-xks-ca" pod "471dfd1b-61fa-43ab-b337-415594a0b4f1" (UID: "471dfd1b-61fa-43ab-b337-415594a0b4f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:00.307384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.307357    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471dfd1b-61fa-43ab-b337-415594a0b4f1-kube-api-access-hchkq" pod "471dfd1b-61fa-43ab-b337-415594a0b4f1" (UID: "471dfd1b-61fa-43ab-b337-415594a0b4f1"). InnerVolumeSpecName "kube-api-access-hchkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:00.406321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.406276    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchkq\" (UniqueName: \"kubernetes.io/projected/471dfd1b-61fa-43ab-b337-415594a0b4f1-kube-api-access-hchkq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:00.406321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.406318    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:00.406321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:00.406330    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471dfd1b-61fa-43ab-b337-415594a0b4f1-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.022780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.022750    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"
Oct 30 20:01:01.022899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.022780    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq" event={"ID":"471dfd1b-61fa-43ab-b337-415594a0b4f1","Type":"ContainerDied","Data":"6beb73d57a427ebf0503abafaa4c8afd5212bd26377c13a9f57623c13ad387ee"}
Oct 30 20:01:01.022899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.022830    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6beb73d57a427ebf0503abafaa4c8afd5212bd26377c13a9f57623c13ad387ee"
Oct 30 20:01:01.412183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.412155    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 20:01:01.448639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448602    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-kube-api-access-wgfkm\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-kube-api-access-wgfkm\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.448798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448696    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-podinfo\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.448798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448730    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-token\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-token\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.448798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448757    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-certs\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.448798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448787    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-data\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.449038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448851    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-socket\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.449038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448886    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00ba69a-0c7c-45da-9e5d-16b26139248b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ba69a-0c7c-45da-9e5d-16b26139248b-istiod-ca-cert\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.449038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.448978    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-credential-socket\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.449200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.449038    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-envoy\") pod \"d00ba69a-0c7c-45da-9e5d-16b26139248b\" (UID: \"d00ba69a-0c7c-45da-9e5d-16b26139248b\") "
Oct 30 20:01:01.449442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.449412    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-credential-socket" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.449442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.449394    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-data" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.449708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.449685    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-socket" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.449784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.449606    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00ba69a-0c7c-45da-9e5d-16b26139248b-istiod-ca-cert" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:01.449926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.449905    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-certs" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.450058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.449940    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.450178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.450157    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.450252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.450183    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ba69a-0c7c-45da-9e5d-16b26139248b-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.450252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.450199    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.451959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.451932    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-kube-api-access-wgfkm" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "kube-api-access-wgfkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:01.452274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.452255    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-token" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:01.452365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.452351    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-envoy" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.452428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.452367    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-podinfo" pod "d00ba69a-0c7c-45da-9e5d-16b26139248b" (UID: "d00ba69a-0c7c-45da-9e5d-16b26139248b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:01.551857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.551813    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgfkm\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-kube-api-access-wgfkm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.551857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.551857    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.552103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.551871    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.552103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.551885    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:01.552103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:01.551897    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ba69a-0c7c-45da-9e5d-16b26139248b-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:02.032541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.032459    4519 generic.go:356] "Generic (PLEG): container finished" podID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerID="c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14" exitCode=0
Oct 30 20:01:02.032541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.032500    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" event={"ID":"d00ba69a-0c7c-45da-9e5d-16b26139248b","Type":"ContainerDied","Data":"c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14"}
Oct 30 20:01:02.032541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.032525    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"
Oct 30 20:01:02.032856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.032571    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q" event={"ID":"d00ba69a-0c7c-45da-9e5d-16b26139248b","Type":"ContainerDied","Data":"3ecfcbf084970918d79322cafa4a4bf6d776ccab5f968f7b1b56fac86d85bedc"}
Oct 30 20:01:02.032856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.032603    4519 scope.go:122] "RemoveContainer" containerID="c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14"
Oct 30 20:01:02.044185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.044166    4519 scope.go:122] "RemoveContainer" containerID="f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0"
Oct 30 20:01:02.056696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.056676    4519 scope.go:122] "RemoveContainer" containerID="c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14"
Oct 30 20:01:02.057039 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:01:02.057013    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14\": container with ID starting with c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14 not found: ID does not exist" containerID="c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14"
Oct 30 20:01:02.057156 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.057045    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14"} err="failed to get container status \"c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14\": rpc error: code = NotFound desc = could not find container \"c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14\": container with ID starting with c56543c52bee4901b2ec34960a52807bcf756ab3b2311034bec29cbc6872bc14 not found: ID does not exist"
Oct 30 20:01:02.057156 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.057065    4519 scope.go:122] "RemoveContainer" containerID="f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0"
Oct 30 20:01:02.057404 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:01:02.057378    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0\": container with ID starting with f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0 not found: ID does not exist" containerID="f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0"
Oct 30 20:01:02.057514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.057408    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0"} err="failed to get container status \"f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0\": rpc error: code = NotFound desc = could not find container \"f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0\": container with ID starting with f748b96f5822d3e4be9c153251042b2114301aab846a25ee94f0a272e13919f0 not found: ID does not exist"
Oct 30 20:01:02.099923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.099890    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"]
Oct 30 20:01:02.111427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:02.111399    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-93263c-openshift-default-d8cd5857c-n6j4q"]
Oct 30 20:01:02.244804 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:01:02.244746    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:01:03.614838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:03.614777    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" path="/var/lib/kubelet/pods/d00ba69a-0c7c-45da-9e5d-16b26139248b/volumes"
Oct 30 20:01:08.451573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:08.451452    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"]
Oct 30 20:01:08.463396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:08.463360    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-cxbg2"]
Oct 30 20:01:09.614288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:09.614249    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f71194d-b774-4e3c-9dfa-1aeef49413fd" path="/var/lib/kubelet/pods/9f71194d-b774-4e3c-9dfa-1aeef49413fd/volumes"
Oct 30 20:01:32.222616 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:01:32.222577    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:01:41.125478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.125426    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"]
Oct 30 20:01:41.126144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126127    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy"
Oct 30 20:01:41.126231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126147    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy"
Oct 30 20:01:41.126231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126164    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy"
Oct 30 20:01:41.126231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126170    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy"
Oct 30 20:01:41.126231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126224    4519 cpu_manager.go:414] "containerMap: removing container" podUID="471dfd1b-61fa-43ab-b337-415594a0b4f1" containerName="revoke-keys"
Oct 30 20:01:41.126231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126230    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471dfd1b-61fa-43ab-b337-415594a0b4f1" containerName="revoke-keys"
Oct 30 20:01:41.126419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126250    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy"
Oct 30 20:01:41.126419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126254    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy"
Oct 30 20:01:41.126419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126264    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy"
Oct 30 20:01:41.126419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126268    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy"
Oct 30 20:01:41.126419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126403    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="471dfd1b-61fa-43ab-b337-415594a0b4f1" containerName="revoke-keys"
Oct 30 20:01:41.126419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126415    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy"
Oct 30 20:01:41.126640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126422    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy"
Oct 30 20:01:41.126640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.126433    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc2d597-f684-4579-ba47-7735c112706a" containerName="istio-proxy"
Oct 30 20:01:41.130598 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.130572    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.148251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.148223    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:41.148663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.148630    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:01:41.157245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.157214    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"]
Oct 30 20:01:41.198629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.198590    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nntm\" (UniqueName: \"kubernetes.io/projected/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-kube-api-access-6nntm\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.198819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.198772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.198892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.198835    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.300269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.300221    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.300503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.300313    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.300616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.300595    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nntm\" (UniqueName: \"kubernetes.io/projected/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-kube-api-access-6nntm\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.301250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.301089    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.317540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.315059    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.325517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.321361    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nntm\" (UniqueName: \"kubernetes.io/projected/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-kube-api-access-6nntm\") pod \"maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.449879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.449839    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:41.759709 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:01:41.759662    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde2e36f_c69d_48b8_84a8_e00c86b0aacd.slice/crio-044336a41460a1171b655165ccacc720244e820e950fe24d6abf9f32a6c28c6b WatchSource:0}: Error finding container 044336a41460a1171b655165ccacc720244e820e950fe24d6abf9f32a6c28c6b: Status 404 returned error can't find the container with id 044336a41460a1171b655165ccacc720244e820e950fe24d6abf9f32a6c28c6b
Oct 30 20:01:41.761738 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.761712    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"]
Oct 30 20:01:41.801867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.801837    4519 scope.go:122] "RemoveContainer" containerID="d46e7761f48d80e0ed336a87c1203b2c4b7140c4bc23d35a36b3e817108eb8ac"
Oct 30 20:01:41.812909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.812881    4519 scope.go:122] "RemoveContainer" containerID="b409047454d51999e63bf505a6e27edd906a457c829771ab9419a8dd4198e93d"
Oct 30 20:01:41.882027 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:41.881985    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc" event={"ID":"fde2e36f-c69d-48b8-84a8-e00c86b0aacd","Type":"ContainerStarted","Data":"044336a41460a1171b655165ccacc720244e820e950fe24d6abf9f32a6c28c6b"}
Oct 30 20:01:42.893105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:42.893053    4519 generic.go:356] "Generic (PLEG): container finished" podID="fde2e36f-c69d-48b8-84a8-e00c86b0aacd" containerID="c3c874d1a808264bf7342faad54cbefa52ae1fd8a7dea6d342b890cdd68d6ccd" exitCode=0
Oct 30 20:01:42.893531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:42.893130    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc" event={"ID":"fde2e36f-c69d-48b8-84a8-e00c86b0aacd","Type":"ContainerDied","Data":"c3c874d1a808264bf7342faad54cbefa52ae1fd8a7dea6d342b890cdd68d6ccd"}
Oct 30 20:01:44.093148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.093115    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:44.251942 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.251897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-xks-ca\") pod \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") "
Oct 30 20:01:44.252215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.252196    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-kube-api-access-6nntm\" (UniqueName: \"kubernetes.io/projected/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-kube-api-access-6nntm\") pod \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") "
Oct 30 20:01:44.252279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.252254    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-openshift-ca\") pod \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\" (UID: \"fde2e36f-c69d-48b8-84a8-e00c86b0aacd\") "
Oct 30 20:01:44.252707 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.252669    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-openshift-ca" pod "fde2e36f-c69d-48b8-84a8-e00c86b0aacd" (UID: "fde2e36f-c69d-48b8-84a8-e00c86b0aacd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:44.253058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.253036    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:44.254325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.254288    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-xks-ca" pod "fde2e36f-c69d-48b8-84a8-e00c86b0aacd" (UID: "fde2e36f-c69d-48b8-84a8-e00c86b0aacd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:44.254531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.254504    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-kube-api-access-6nntm" pod "fde2e36f-c69d-48b8-84a8-e00c86b0aacd" (UID: "fde2e36f-c69d-48b8-84a8-e00c86b0aacd"). InnerVolumeSpecName "kube-api-access-6nntm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:44.355096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.355051    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:44.355096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.355093    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nntm\" (UniqueName: \"kubernetes.io/projected/fde2e36f-c69d-48b8-84a8-e00c86b0aacd-kube-api-access-6nntm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:01:44.948937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.944653    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc" event={"ID":"fde2e36f-c69d-48b8-84a8-e00c86b0aacd","Type":"ContainerDied","Data":"044336a41460a1171b655165ccacc720244e820e950fe24d6abf9f32a6c28c6b"}
Oct 30 20:01:44.948937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.944712    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="044336a41460a1171b655165ccacc720244e820e950fe24d6abf9f32a6c28c6b"
Oct 30 20:01:44.948937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:44.944770    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"
Oct 30 20:01:57.709374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.709324    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"]
Oct 30 20:01:57.710016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.709997    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fde2e36f-c69d-48b8-84a8-e00c86b0aacd" containerName="revoke-keys"
Oct 30 20:01:57.710176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.710157    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde2e36f-c69d-48b8-84a8-e00c86b0aacd" containerName="revoke-keys"
Oct 30 20:01:57.710370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.710355    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00ba69a-0c7c-45da-9e5d-16b26139248b" containerName="istio-proxy"
Oct 30 20:01:57.710432 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.710373    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde2e36f-c69d-48b8-84a8-e00c86b0aacd" containerName="revoke-keys"
Oct 30 20:01:57.715632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.715597    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"]
Oct 30 20:01:57.715769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.715728    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.782820 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.782767    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-936f70-openshift-default-dockercfg-jtq7n\""
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.882027    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-credential-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.882174    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-envoy\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.882505    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-podinfo\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.882878    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.883221    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-certs\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.883893    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istiod-ca-cert\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.884245    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrq2\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-kube-api-access-hnrq2\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.884292    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-data\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:57.888362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:57.884360    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-token\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.016550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.013557    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istiod-ca-cert\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.036907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.036869    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrq2\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-kube-api-access-hnrq2\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.038191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.038155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-data\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.038448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.038427    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-token\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.038791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.038770    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-credential-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.038920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.038907    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-envoy\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.039074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.039061    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-podinfo\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.039215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.039199    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.039327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.039312    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-certs\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.040072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.014488    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istiod-ca-cert\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.044569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.041025    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-data\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.052907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.045621    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-credential-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.053397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.046768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-socket\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.057552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.047344    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-certs\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.066674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.059270    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-envoy\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.066674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.066155    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrq2\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-kube-api-access-hnrq2\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.079486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.073111    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-token\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.111492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.107924    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-podinfo\") pod \"e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") " pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.338595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.338505    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:58.689833 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:01:58.689775    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f9b0a1_0b9e_4287_9210_89ff7edd7e8e.slice/crio-1ab7a6883bb773c28db3c0394651e4da210d2ce06f7ecb95aeaf5eac924e505b WatchSource:0}: Error finding container 1ab7a6883bb773c28db3c0394651e4da210d2ce06f7ecb95aeaf5eac924e505b: Status 404 returned error can't find the container with id 1ab7a6883bb773c28db3c0394651e4da210d2ce06f7ecb95aeaf5eac924e505b
Oct 30 20:01:58.690102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.690071    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"]
Oct 30 20:01:58.693559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.693517    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:01:58.693676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.693608    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:01:58.693676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:58.693658    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:01:59.188880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:59.188831    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" event={"ID":"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e","Type":"ContainerStarted","Data":"a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3"}
Oct 30 20:01:59.189406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:59.188940    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" event={"ID":"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e","Type":"ContainerStarted","Data":"1ab7a6883bb773c28db3c0394651e4da210d2ce06f7ecb95aeaf5eac924e505b"}
Oct 30 20:01:59.262882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:59.262813    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podStartSLOduration=2.26279231 podStartE2EDuration="2.26279231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:59.246924298 +0000 UTC m=+1643.747628023" watchObservedRunningTime="2026-10-30 20:01:59.26279231 +0000 UTC m=+1643.763496041"
Oct 30 20:01:59.339438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:59.339399    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:01:59.345454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:01:59.345422    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:02:00.198113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:00.198072    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:02:00.199512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:00.199449    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:02:02.369922 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:02:02.369879    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:02:05.099057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:05.099009    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:08.572627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:08.572531    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"]
Oct 30 20:02:08.919496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:08.919446    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"]
Oct 30 20:02:08.927538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:08.919814    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:08.960499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:08.959706    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7b5635-openshift-default-dockercfg-6r2j2\""
Oct 30 20:02:09.069579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.069535    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64c028c-e19b-4191-b338-9b863873c47b-istio-podinfo\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.069998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.069970    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-credential-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.074637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.074608    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-envoy\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.075182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.075162    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64c028c-e19b-4191-b338-9b863873c47b-istiod-ca-cert\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.080581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.080552    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-data\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.080793 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.080772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznjf\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-kube-api-access-vznjf\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.080920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.080904    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-certs\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.081160 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.081142    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.081279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.081264    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-istio-token\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.120658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.120565    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:09.184379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184273    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-envoy\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184412    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64c028c-e19b-4191-b338-9b863873c47b-istiod-ca-cert\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184445    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-data\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184513    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznjf\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-kube-api-access-vznjf\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184541    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-certs\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184655    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184683    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-istio-token\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184915 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184809    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64c028c-e19b-4191-b338-9b863873c47b-istio-podinfo\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.184915 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.184904    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-credential-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.185460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.185435    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-credential-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.186188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.186153    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-certs\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.188376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.187097    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64c028c-e19b-4191-b338-9b863873c47b-istiod-ca-cert\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.188526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.187373    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-data\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.188728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.188313    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-socket\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.213988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.213957    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64c028c-e19b-4191-b338-9b863873c47b-istio-podinfo\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.238548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.227334    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznjf\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-kube-api-access-vznjf\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.238752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.227389    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:09.240063 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.227576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:09.241185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.241152    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:09.265483 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.255327    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-envoy\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.265483 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.264896    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-istio-token\") pod \"e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") " pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.277096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.276327    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:09.933116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.933082    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"]
Oct 30 20:02:09.933907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.933866    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:09.934017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.933940    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:09.934017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:09.933972    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:10.979817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:10.979763    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" event={"ID":"d64c028c-e19b-4191-b338-9b863873c47b","Type":"ContainerStarted","Data":"f8a69c3aa3a260f493591da645463e2fe8252be6d12a248cdefb5dd5c74ed391"}
Oct 30 20:02:10.980456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:10.980415    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" event={"ID":"d64c028c-e19b-4191-b338-9b863873c47b","Type":"ContainerStarted","Data":"e74c39370c467a49982371eac6ffbf76ab936cb639dc0a116c243e4505c98397"}
Oct 30 20:02:11.092438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:11.092298    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podStartSLOduration=3.092272475 podStartE2EDuration="3.092272475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:11.066782079 +0000 UTC m=+1655.567485807" watchObservedRunningTime="2026-10-30 20:02:11.092272475 +0000 UTC m=+1655.592976201"
Oct 30 20:02:11.333489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:11.291549    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:11.364555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:11.342572    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:11.997123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:11.996816    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:11.998264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:11.998237    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:02:13.851493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:13.851440    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"]
Oct 30 20:02:13.864149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:13.864117    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d96913-rwpb8"]
Oct 30 20:02:15.618337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:15.618278    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40857ad-a571-4697-95ec-4c3be56da5c1" path="/var/lib/kubelet/pods/e40857ad-a571-4697-95ec-4c3be56da5c1/volumes"
Oct 30 20:02:15.620610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:15.620581    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:02:15.620771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:15.620627    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:16.874817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:16.874407    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": read tcp 10.133.0.2:58304->10.133.0.185:15021: read: connection reset by peer" start-of-body=
Oct 30 20:02:16.874817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:16.874525    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": read tcp 10.133.0.2:58304->10.133.0.185:15021: read: connection reset by peer"
Oct 30 20:02:17.380990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.380956    4519 generic.go:356] "Generic (PLEG): container finished" podID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerID="bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f" exitCode=0
Oct 30 20:02:17.381178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.381029    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" event={"ID":"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37","Type":"ContainerDied","Data":"bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f"}
Oct 30 20:02:17.381705 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.381681    4519 scope.go:122] "RemoveContainer" containerID="bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f"
Oct 30 20:02:17.384030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.384008    4519 generic.go:356] "Generic (PLEG): container finished" podID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerID="0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405" exitCode=0
Oct 30 20:02:17.384132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.384088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" event={"ID":"11cf99a9-cef2-4e0b-8455-47bbd2c33edf","Type":"ContainerDied","Data":"0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405"}
Oct 30 20:02:17.384265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.384232    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:17.384348 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.384331    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:17.384395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.384382    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:17.384595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.384578    4519 scope.go:122] "RemoveContainer" containerID="0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405"
Oct 30 20:02:17.389414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.389314    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:17.389414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.389379    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:17.389553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:17.389416    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:18.414941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.414898    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" event={"ID":"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37","Type":"ContainerStarted","Data":"377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4"}
Oct 30 20:02:18.417327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.417301    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" event={"ID":"11cf99a9-cef2-4e0b-8455-47bbd2c33edf","Type":"ContainerStarted","Data":"e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba"}
Oct 30 20:02:18.870604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.869627    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 20:02:18.870604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.869802    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.870604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.869850    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:18.967490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.964871    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:02:18.967490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.964999    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.967490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:18.965041    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 20:02:19.868029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:19.863375    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.868029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:19.863947    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:19.972491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:19.971955    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.972491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:19.972048    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 20:02:20.885495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:20.884663    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:20.885495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:20.884749    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:20.977491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:20.972026    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 20:02:20.977491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:20.972090    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 20:02:21.920363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:21.919607    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:21.920363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:21.919700    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:21.961495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:21.960155    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 20:02:21.961495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:21.960231    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 20:02:22.071493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.065962    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:22.106498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.103533    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:22.118497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.115051    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:22.128500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.125794    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:22.129489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.128841    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 20:02:22.137654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.137611    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 20:02:22.193556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.154651    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:22.203503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.197647    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:22.735572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.735528    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"]
Oct 30 20:02:22.742152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.742098    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-540a43-5b4nq"]
Oct 30 20:02:22.880561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.874619    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:22.880561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.874706    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:22.975502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.968984    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 20:02:22.975502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:22.969046    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 20:02:23.031565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.030914    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"]
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.098395    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.118599    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c17154-openshift-default-dockercfg-9rq94\""
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149273    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istiod-ca-cert\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149386    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-data\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149415    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-envoy\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149541    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhbn\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-kube-api-access-whhbn\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149604    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-certs\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149657    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-podinfo\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149733    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-credential-socket\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149784    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-token\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.166294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.149852    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-socket\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.242167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.242124    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"]
Oct 30 20:02:23.264505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.258481    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-credential-socket\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.265292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.265144    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-credential-socket\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.275592 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.265516    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-token\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.275973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.275927    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-socket\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.276629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.276598    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istiod-ca-cert\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.409950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.285238    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istiod-ca-cert\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.409950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.276509    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-socket\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.409950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.334050    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-token\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.448557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.448124    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-data\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.453898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.449149    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-data\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.453898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.449224    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-envoy\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.453898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.449444    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhbn\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-kube-api-access-whhbn\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.453898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.450232    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-certs\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.453898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.450413    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-podinfo\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.453898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.453851    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-certs\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.505230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.504820    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-envoy\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.517678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.517598    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhbn\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-kube-api-access-whhbn\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.587772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.581208    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-podinfo\") pod \"e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") " pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.667493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.666415    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471dfd1b-61fa-43ab-b337-415594a0b4f1" path="/var/lib/kubelet/pods/471dfd1b-61fa-43ab-b337-415594a0b4f1/volumes"
Oct 30 20:02:23.856490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.856140    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:23.877506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.862880    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:23.877506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.862942    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:23.965498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.964884    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 20:02:23.965498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:23.964952    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 20:02:24.545500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.545437    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"]
Oct 30 20:02:24.550521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.549078    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:24.550521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.549161    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:24.550521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.549203    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:24.865557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.863543    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:24.865557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.863627    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:24.942693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.942641    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" event={"ID":"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8","Type":"ContainerStarted","Data":"d8deb4449947ace196ed08c659ba81f71f59e3414568f8e277dcb10aae0c3cd4"}
Oct 30 20:02:24.942856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.942769    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" event={"ID":"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8","Type":"ContainerStarted","Data":"9b73c743430345df9f63e9d0c973c9fb0fb48457d6cd91e7fde6ab7d7e00bb4e"}
Oct 30 20:02:24.985492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.963659    4519 patch_prober.go:29] interesting pod/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 20:02:24.985492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:24.963756    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 20:02:25.111498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:25.110619    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podStartSLOduration=3.110593325 podStartE2EDuration="3.110593325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:25.082845369 +0000 UTC m=+1669.583549102" watchObservedRunningTime="2026-10-30 20:02:25.110593325 +0000 UTC m=+1669.611297063"
Oct 30 20:02:25.862491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:25.861501    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:25.862491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:25.861896    4519 patch_prober.go:29] interesting pod/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:25.862491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:25.861964    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 20:02:25.863126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:25.862629    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:02:25.863126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:25.862687    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:02:26.082246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.081803    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:02:26.084375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.084222    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:02:26.094878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.094804    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:02:26.864322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.864290    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:26.865262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.865241    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 20:02:26.865624 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.865596    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 20:02:26.866460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.866439    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 20:02:26.958981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.958946    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:26.959816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:26.959798    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:02:32.264811 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:02:32.264758    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:02:33.055255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.055211    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"]
Oct 30 20:02:33.066294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.066263    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.106543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.106501    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-istio-token\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.106724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.106551    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a28953-5648-4dad-8d3e-fe423819340f-istiod-ca-cert\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.106724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.106700    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-certs\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.106860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.106736    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-data\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.106860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.106802    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-envoy\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.107065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.107034    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-credential-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.107166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.107148    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.107236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.107221    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a28953-5648-4dad-8d3e-fe423819340f-istio-podinfo\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.107309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.107294    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4k9\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-kube-api-access-dv4k9\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.109080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.109038    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ebe57b-openshift-default-dockercfg-556nv\""
Oct 30 20:02:33.113796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.113762    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"]
Oct 30 20:02:33.208655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.208609    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.208861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.208713    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a28953-5648-4dad-8d3e-fe423819340f-istio-podinfo\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.208861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.208787    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4k9\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-kube-api-access-dv4k9\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.208861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.208826    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-istio-token\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.208861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.208858    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a28953-5648-4dad-8d3e-fe423819340f-istiod-ca-cert\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.209048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209018    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-certs\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.209102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209055    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-data\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.209159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209120    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-envoy\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.210042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209255    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-credential-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.210042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209341    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.210042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209593    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-certs\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.210042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209713    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-data\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.210042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209802    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a28953-5648-4dad-8d3e-fe423819340f-istiod-ca-cert\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.210042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.209869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-credential-socket\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.212900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.212866    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-envoy\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.213041 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.212953    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a28953-5648-4dad-8d3e-fe423819340f-istio-podinfo\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.231211 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.231169    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4k9\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-kube-api-access-dv4k9\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.232228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.232201    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-istio-token\") pod \"e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") " pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:33.385969 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:33.385863    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:34.048595 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:02:34.048558    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a28953_5648_4dad_8d3e_fe423819340f.slice/crio-b5cc4aed87b901b4cfb05197ef6e2f8d04fe5a0229d269eb957facfccd103982 WatchSource:0}: Error finding container b5cc4aed87b901b4cfb05197ef6e2f8d04fe5a0229d269eb957facfccd103982: Status 404 returned error can't find the container with id b5cc4aed87b901b4cfb05197ef6e2f8d04fe5a0229d269eb957facfccd103982
Oct 30 20:02:34.049898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.049868    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"]
Oct 30 20:02:34.052710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.052670    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:34.052823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.052753    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:34.052823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.052783    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:34.231829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.231088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" event={"ID":"f4a28953-5648-4dad-8d3e-fe423819340f","Type":"ContainerStarted","Data":"2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25"}
Oct 30 20:02:34.231829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.231162    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" event={"ID":"f4a28953-5648-4dad-8d3e-fe423819340f","Type":"ContainerStarted","Data":"b5cc4aed87b901b4cfb05197ef6e2f8d04fe5a0229d269eb957facfccd103982"}
Oct 30 20:02:34.274879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.274808    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podStartSLOduration=1.2747906580000001 podStartE2EDuration="1.274790658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:34.27460768 +0000 UTC m=+1678.775311406" watchObservedRunningTime="2026-10-30 20:02:34.274790658 +0000 UTC m=+1678.775494382"
Oct 30 20:02:34.393555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:34.392598    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:35.399281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:35.399231    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:36.250831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:36.250788    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:36.252040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:36.252007    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:02:37.682546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.682484    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"]
Oct 30 20:02:37.688504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.688425    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.702796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.702759    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"]
Oct 30 20:02:37.784671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.784625    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:37.784854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.784737    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:02:37.838801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.838755    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.839093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.839070    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2sd\" (UniqueName: \"kubernetes.io/projected/654987e5-b9d2-452a-b2b1-2235a06e4a9f-kube-api-access-jm2sd\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.839256 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.839235    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.941026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.940929    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.941178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.941164    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2sd\" (UniqueName: \"kubernetes.io/projected/654987e5-b9d2-452a-b2b1-2235a06e4a9f-kube-api-access-jm2sd\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.941287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.941274    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.941625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.941567    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.943928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.943880    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:37.962529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:37.962449    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2sd\" (UniqueName: \"kubernetes.io/projected/654987e5-b9d2-452a-b2b1-2235a06e4a9f-kube-api-access-jm2sd\") pod \"maas-api-revoke-keys-e2e-mt-2224e496-2xcql\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:38.006607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:38.006564    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:38.340744 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:02:38.340696    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654987e5_b9d2_452a_b2b1_2235a06e4a9f.slice/crio-0822984e8a49101c184da05d39c3f693f6ad4c4a098b5a43efa762709a759a8f WatchSource:0}: Error finding container 0822984e8a49101c184da05d39c3f693f6ad4c4a098b5a43efa762709a759a8f: Status 404 returned error can't find the container with id 0822984e8a49101c184da05d39c3f693f6ad4c4a098b5a43efa762709a759a8f
Oct 30 20:02:38.344566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:38.344536    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"]
Oct 30 20:02:39.301010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:39.300960    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql" event={"ID":"654987e5-b9d2-452a-b2b1-2235a06e4a9f","Type":"ContainerStarted","Data":"c9885dfc37087e1afa46c4e107ef484a070309d081a26d472d7ceefaaae9b283"}
Oct 30 20:02:39.301443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:39.301033    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql" event={"ID":"654987e5-b9d2-452a-b2b1-2235a06e4a9f","Type":"ContainerStarted","Data":"0822984e8a49101c184da05d39c3f693f6ad4c4a098b5a43efa762709a759a8f"}
Oct 30 20:02:39.343235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:39.343173    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql" podStartSLOduration=2.343156302 podStartE2EDuration="2.343156302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:39.340433482 +0000 UTC m=+1683.841137210" watchObservedRunningTime="2026-10-30 20:02:39.343156302 +0000 UTC m=+1683.843860026"
Oct 30 20:02:40.310721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:40.310682    4519 generic.go:356] "Generic (PLEG): container finished" podID="654987e5-b9d2-452a-b2b1-2235a06e4a9f" containerID="c9885dfc37087e1afa46c4e107ef484a070309d081a26d472d7ceefaaae9b283" exitCode=0
Oct 30 20:02:40.311164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:40.310762    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql" event={"ID":"654987e5-b9d2-452a-b2b1-2235a06e4a9f","Type":"ContainerDied","Data":"c9885dfc37087e1afa46c4e107ef484a070309d081a26d472d7ceefaaae9b283"}
Oct 30 20:02:41.509835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.509805    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:41.674119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.674086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654987e5-b9d2-452a-b2b1-2235a06e4a9f-kube-api-access-jm2sd\" (UniqueName: \"kubernetes.io/projected/654987e5-b9d2-452a-b2b1-2235a06e4a9f-kube-api-access-jm2sd\") pod \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") "
Oct 30 20:02:41.674541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.674515    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-xks-ca\") pod \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") "
Oct 30 20:02:41.674630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.674571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-openshift-ca\") pod \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\" (UID: \"654987e5-b9d2-452a-b2b1-2235a06e4a9f\") "
Oct 30 20:02:41.674992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.674966    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-openshift-ca" pod "654987e5-b9d2-452a-b2b1-2235a06e4a9f" (UID: "654987e5-b9d2-452a-b2b1-2235a06e4a9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:41.675307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.675281    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:02:41.676688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.676661    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654987e5-b9d2-452a-b2b1-2235a06e4a9f-kube-api-access-jm2sd" pod "654987e5-b9d2-452a-b2b1-2235a06e4a9f" (UID: "654987e5-b9d2-452a-b2b1-2235a06e4a9f"). InnerVolumeSpecName "kube-api-access-jm2sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:41.677161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.677129    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-xks-ca" pod "654987e5-b9d2-452a-b2b1-2235a06e4a9f" (UID: "654987e5-b9d2-452a-b2b1-2235a06e4a9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:41.777181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.777132    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2sd\" (UniqueName: \"kubernetes.io/projected/654987e5-b9d2-452a-b2b1-2235a06e4a9f-kube-api-access-jm2sd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:02:41.777181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:41.777171    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654987e5-b9d2-452a-b2b1-2235a06e4a9f-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:02:42.330756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:42.330726    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"
Oct 30 20:02:42.330974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:42.330763    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql" event={"ID":"654987e5-b9d2-452a-b2b1-2235a06e4a9f","Type":"ContainerDied","Data":"0822984e8a49101c184da05d39c3f693f6ad4c4a098b5a43efa762709a759a8f"}
Oct 30 20:02:42.330974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:42.330825    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0822984e8a49101c184da05d39c3f693f6ad4c4a098b5a43efa762709a759a8f"
Oct 30 20:02:56.210832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:56.210788    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"]
Oct 30 20:02:56.224895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:56.223491    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa42c3-sq2wc"]
Oct 30 20:02:57.617397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:02:57.617364    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde2e36f-c69d-48b8-84a8-e00c86b0aacd" path="/var/lib/kubelet/pods/fde2e36f-c69d-48b8-84a8-e00c86b0aacd/volumes"
Oct 30 20:03:02.225861 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:03:02.225813    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:03:04.897862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.897821    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"]
Oct 30 20:03:04.898580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.898563    4519 cpu_manager.go:414] "containerMap: removing container" podUID="654987e5-b9d2-452a-b2b1-2235a06e4a9f" containerName="revoke-keys"
Oct 30 20:03:04.898641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.898581    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654987e5-b9d2-452a-b2b1-2235a06e4a9f" containerName="revoke-keys"
Oct 30 20:03:04.898718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.898708    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="654987e5-b9d2-452a-b2b1-2235a06e4a9f" containerName="revoke-keys"
Oct 30 20:03:04.902364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.902346    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:04.919649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.919615    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"]
Oct 30 20:03:04.951562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.951455    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:03:04.951800 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:04.951460    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:03:05.009738 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.009700    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4vk\" (UniqueName: \"kubernetes.io/projected/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-kube-api-access-jq4vk\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.009920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.009759    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.009970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.009949    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.111626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.111584    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4vk\" (UniqueName: \"kubernetes.io/projected/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-kube-api-access-jq4vk\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.111826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.111734    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.111879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.111846    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.112481 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.112436    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.114603 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.114575    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.131187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.131145    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4vk\" (UniqueName: \"kubernetes.io/projected/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-kube-api-access-jq4vk\") pod \"maas-api-revoke-keys-e2e-api-b-652137-7hxwb\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.218049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.218007    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:05.532315 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:03:05.532261    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d24f1e_023a_48b4_bcf4_7e1b9a3747a1.slice/crio-f4841d2d766026898ed31ff44bfaa9b7c73fe678e06abbc4fd7acecf0c169ea4 WatchSource:0}: Error finding container f4841d2d766026898ed31ff44bfaa9b7c73fe678e06abbc4fd7acecf0c169ea4: Status 404 returned error can't find the container with id f4841d2d766026898ed31ff44bfaa9b7c73fe678e06abbc4fd7acecf0c169ea4
Oct 30 20:03:05.534324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.534301    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"]
Oct 30 20:03:05.658311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:05.658265    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb" event={"ID":"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1","Type":"ContainerStarted","Data":"f4841d2d766026898ed31ff44bfaa9b7c73fe678e06abbc4fd7acecf0c169ea4"}
Oct 30 20:03:06.669802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:06.669764    4519 generic.go:356] "Generic (PLEG): container finished" podID="10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" containerID="ce12c59207d0e435a722225469195a77aa7e2bc93eaf91f8ca5fbcb8a4cebdae" exitCode=0
Oct 30 20:03:06.670209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:06.669888    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb" event={"ID":"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1","Type":"ContainerDied","Data":"ce12c59207d0e435a722225469195a77aa7e2bc93eaf91f8ca5fbcb8a4cebdae"}
Oct 30 20:03:07.884482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:07.884437    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:08.004671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.004565    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-openshift-ca\") pod \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") "
Oct 30 20:03:08.004905 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.004884    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-kube-api-access-jq4vk\" (UniqueName: \"kubernetes.io/projected/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-kube-api-access-jq4vk\") pod \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") "
Oct 30 20:03:08.005005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.004955    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-openshift-ca" pod "10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" (UID: "10d24f1e-023a-48b4-bcf4-7e1b9a3747a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:08.005073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.005012    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-xks-ca\") pod \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\" (UID: \"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1\") "
Oct 30 20:03:08.005745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.005720    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:08.007240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.007208    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-xks-ca" pod "10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" (UID: "10d24f1e-023a-48b4-bcf4-7e1b9a3747a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:08.007240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.007222    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-kube-api-access-jq4vk" pod "10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" (UID: "10d24f1e-023a-48b4-bcf4-7e1b9a3747a1"). InnerVolumeSpecName "kube-api-access-jq4vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:08.107163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.107126    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4vk\" (UniqueName: \"kubernetes.io/projected/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-kube-api-access-jq4vk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:08.107163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.107157    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:08.694860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.694829    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"
Oct 30 20:03:08.695082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.694829    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb" event={"ID":"10d24f1e-023a-48b4-bcf4-7e1b9a3747a1","Type":"ContainerDied","Data":"f4841d2d766026898ed31ff44bfaa9b7c73fe678e06abbc4fd7acecf0c169ea4"}
Oct 30 20:03:08.695082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:08.694949    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4841d2d766026898ed31ff44bfaa9b7c73fe678e06abbc4fd7acecf0c169ea4"
Oct 30 20:03:14.891024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:14.890991    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"]
Oct 30 20:03:14.891703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:14.891671    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy" containerID="cri-o://bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c" gracePeriod=30
Oct 30 20:03:14.984895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:14.984856    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"]
Oct 30 20:03:14.989893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:14.989861    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2224e496-2xcql"]
Oct 30 20:03:15.345676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:15.345640    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"]
Oct 30 20:03:15.345884 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:15.345863    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" containerID="cri-o://f7d5e8060e4fc9ca0dfa563fe49fccbce188bfe06e212fdce8f9338004c21d5f" gracePeriod=30
Oct 30 20:03:15.632978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:15.632681    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654987e5-b9d2-452a-b2b1-2235a06e4a9f" path="/var/lib/kubelet/pods/654987e5-b9d2-452a-b2b1-2235a06e4a9f/volumes"
Oct 30 20:03:20.299840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.299799    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319343    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-credential-socket\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319480    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454c8916-e204-493e-9650-88be170d9224-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454c8916-e204-493e-9650-88be170d9224-istiod-ca-cert\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319523    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-envoy\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319618    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-data\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319758    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-certs\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319814    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-kube-api-access-qpflf\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-kube-api-access-qpflf\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319866    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-istio-token\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-istio-token\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319904    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/454c8916-e204-493e-9650-88be170d9224-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454c8916-e204-493e-9650-88be170d9224-istio-podinfo\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.323078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.319947    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-socket\") pod \"454c8916-e204-493e-9650-88be170d9224\" (UID: \"454c8916-e204-493e-9650-88be170d9224\") "
Oct 30 20:03:20.326502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.325969    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454c8916-e204-493e-9650-88be170d9224-istiod-ca-cert" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:20.326502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.326212    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-credential-socket" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:20.327002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.326965    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-certs" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:20.327194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.327166    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-socket" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:20.327680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.327653    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-data" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:20.333888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.332439    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-istio-token" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:20.333888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.333242    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-envoy" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:20.365933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.365125    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"]
Oct 30 20:03:20.365933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.365386    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl" podUID="8f5a79d6-7154-4e61-a0de-d5886a1e6726" containerName="discovery" containerID="cri-o://c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5" gracePeriod=30
Oct 30 20:03:20.377251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.377206    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-kube-api-access-qpflf" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "kube-api-access-qpflf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:20.390176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.390116    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/454c8916-e204-493e-9650-88be170d9224-istio-podinfo" pod "454c8916-e204-493e-9650-88be170d9224" (UID: "454c8916-e204-493e-9650-88be170d9224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422323    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422357    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422367    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454c8916-e204-493e-9650-88be170d9224-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422375    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422383    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422392    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454c8916-e204-493e-9650-88be170d9224-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422400    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpflf\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-kube-api-access-qpflf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422409    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454c8916-e204-493e-9650-88be170d9224-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.422635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.422419    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454c8916-e204-493e-9650-88be170d9224-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.700415 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.700362    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 20:03:20.735854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.735817    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-kubeconfig\") pod \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") "
Oct 30 20:03:20.736063 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.735931    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-dns-cert\") pod \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") "
Oct 30 20:03:20.736063 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.736003    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-token\") pod \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") "
Oct 30 20:03:20.736063 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.736058    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a79d6-7154-4e61-a0de-d5886a1e6726-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a79d6-7154-4e61-a0de-d5886a1e6726-local-certs\") pod \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") "
Oct 30 20:03:20.736437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.736414    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-kube-api-access-sxbcp\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-kube-api-access-sxbcp\") pod \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") "
Oct 30 20:03:20.736638 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.736616    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-ca-configmap\") pod \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") "
Oct 30 20:03:20.737063 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.737017    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-ca-configmap" pod "8f5a79d6-7154-4e61-a0de-d5886a1e6726" (UID: "8f5a79d6-7154-4e61-a0de-d5886a1e6726"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:20.737420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.737398    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-cacerts\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-cacerts\") pod \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\" (UID: \"8f5a79d6-7154-4e61-a0de-d5886a1e6726\") "
Oct 30 20:03:20.738753 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.738725    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-dns-cert" pod "8f5a79d6-7154-4e61-a0de-d5886a1e6726" (UID: "8f5a79d6-7154-4e61-a0de-d5886a1e6726"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:20.738854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.738801    4519 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-ca-configmap\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.738854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.738835    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-kubeconfig" pod "8f5a79d6-7154-4e61-a0de-d5886a1e6726" (UID: "8f5a79d6-7154-4e61-a0de-d5886a1e6726"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:20.739072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.739051    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-token" pod "8f5a79d6-7154-4e61-a0de-d5886a1e6726" (UID: "8f5a79d6-7154-4e61-a0de-d5886a1e6726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:20.739206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.739185    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-kube-api-access-sxbcp" pod "8f5a79d6-7154-4e61-a0de-d5886a1e6726" (UID: "8f5a79d6-7154-4e61-a0de-d5886a1e6726"). InnerVolumeSpecName "kube-api-access-sxbcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:20.739723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.739674    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-cacerts" pod "8f5a79d6-7154-4e61-a0de-d5886a1e6726" (UID: "8f5a79d6-7154-4e61-a0de-d5886a1e6726"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:20.740145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.740125    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a79d6-7154-4e61-a0de-d5886a1e6726-local-certs" pod "8f5a79d6-7154-4e61-a0de-d5886a1e6726" (UID: "8f5a79d6-7154-4e61-a0de-d5886a1e6726"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:20.840815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.840769    4519 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-csr-dns-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.840815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.840807    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.840815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.840818    4519 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a79d6-7154-4e61-a0de-d5886a1e6726-local-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.841084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.840827    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbcp\" (UniqueName: \"kubernetes.io/projected/8f5a79d6-7154-4e61-a0de-d5886a1e6726-kube-api-access-sxbcp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.841084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.840837    4519 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-cacerts\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.841084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.840848    4519 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f5a79d6-7154-4e61-a0de-d5886a1e6726-istio-kubeconfig\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.942757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.942715    4519 generic.go:356] "Generic (PLEG): container finished" podID="454c8916-e204-493e-9650-88be170d9224" containerID="bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c" exitCode=0
Oct 30 20:03:20.942955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.942768    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"
Oct 30 20:03:20.942955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.942857    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" event={"ID":"454c8916-e204-493e-9650-88be170d9224","Type":"ContainerDied","Data":"bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c"}
Oct 30 20:03:20.942955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.942923    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2" event={"ID":"454c8916-e204-493e-9650-88be170d9224","Type":"ContainerDied","Data":"384652320e766e2b2f734858125977c8ffcee649998cc4363452082b7abc4c1e"}
Oct 30 20:03:20.942955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.942956    4519 scope.go:122] "RemoveContainer" containerID="bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c"
Oct 30 20:03:20.950116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.949868    4519 generic.go:356] "Generic (PLEG): container finished" podID="8f5a79d6-7154-4e61-a0de-d5886a1e6726" containerID="c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5" exitCode=0
Oct 30 20:03:20.950116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.949954    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"
Oct 30 20:03:20.950116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.950042    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl" event={"ID":"8f5a79d6-7154-4e61-a0de-d5886a1e6726","Type":"ContainerDied","Data":"c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5"}
Oct 30 20:03:20.950116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.950083    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl" event={"ID":"8f5a79d6-7154-4e61-a0de-d5886a1e6726","Type":"ContainerDied","Data":"343120a0e295be9c7c8ae5b1b83925d62dd4b4f37c41f3b3eee53d5df0feb6ee"}
Oct 30 20:03:20.954070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.953880    4519 generic.go:356] "Generic (PLEG): container finished" podID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerID="f7d5e8060e4fc9ca0dfa563fe49fccbce188bfe06e212fdce8f9338004c21d5f" exitCode=0
Oct 30 20:03:20.954070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.953949    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" event={"ID":"64e26568-997a-4e1b-a9c3-ec1a57085e7f","Type":"ContainerDied","Data":"f7d5e8060e4fc9ca0dfa563fe49fccbce188bfe06e212fdce8f9338004c21d5f"}
Oct 30 20:03:20.966342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.966187    4519 scope.go:122] "RemoveContainer" containerID="bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c"
Oct 30 20:03:20.975836 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:03:20.975649    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c\": container with ID starting with bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c not found: ID does not exist" containerID="bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c"
Oct 30 20:03:20.975836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.975711    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c"} err="failed to get container status \"bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c\": rpc error: code = NotFound desc = could not find container \"bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c\": container with ID starting with bc97ed50590693347a9cf2090f3e36e7a4e0afd2df8cb37cbd53f272c3ce328c not found: ID does not exist"
Oct 30 20:03:20.975836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:20.975737    4519 scope.go:122] "RemoveContainer" containerID="c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5"
Oct 30 20:03:21.009438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.009406    4519 scope.go:122] "RemoveContainer" containerID="c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5"
Oct 30 20:03:21.009896 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:03:21.009869    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5\": container with ID starting with c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5 not found: ID does not exist" containerID="c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5"
Oct 30 20:03:21.010009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.009910    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5"} err="failed to get container status \"c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5\": rpc error: code = NotFound desc = could not find container \"c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5\": container with ID starting with c08d09083ad5c7a82ff1f6059db1a02a752ecb9975db5aba819e53939104c3e5 not found: ID does not exist"
Oct 30 20:03:21.018367 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.018341    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"]
Oct 30 20:03:21.034695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.029892    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-gk9w2"]
Oct 30 20:03:21.072357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.072322    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"]
Oct 30 20:03:21.077854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.077824    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cggfl"]
Oct 30 20:03:21.097530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.097461    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 20:03:21.150611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.150578    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istiod-ca-cert\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.150955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.150930    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-credential-socket\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.151236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.151215    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-podinfo\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.151385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.151367    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-data\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.151541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.151522    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-token\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-token\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.151820 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.151800    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-socket\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.151965 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.151948    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-certs\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.152123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.152107    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-kube-api-access-x744m\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-kube-api-access-x744m\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.152265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.152252    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-envoy\") pod \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\" (UID: \"64e26568-997a-4e1b-a9c3-ec1a57085e7f\") "
Oct 30 20:03:21.153148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.153123    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-data" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:21.153485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.153446    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istiod-ca-cert" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:21.153878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.153850    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-socket" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:21.154098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.154076    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-credential-socket" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:21.154396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.154372    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-certs" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:21.157836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.157812    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-podinfo" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:21.159547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.158723    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-envoy" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:21.159855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.159811    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-token" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:21.161078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.161053    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-kube-api-access-x744m" pod "64e26568-997a-4e1b-a9c3-ec1a57085e7f" (UID: "64e26568-997a-4e1b-a9c3-ec1a57085e7f"). InnerVolumeSpecName "kube-api-access-x744m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:21.261803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.261573    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.261803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.261612    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.261803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.261624    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.261803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.261639    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.262230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.261651    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.262230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.262100    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x744m\" (UniqueName: \"kubernetes.io/projected/64e26568-997a-4e1b-a9c3-ec1a57085e7f-kube-api-access-x744m\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.262230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.262120    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.262230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.262135    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e26568-997a-4e1b-a9c3-ec1a57085e7f-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.262230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.262150    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e26568-997a-4e1b-a9c3-ec1a57085e7f-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:03:21.614531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.614398    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454c8916-e204-493e-9650-88be170d9224" path="/var/lib/kubelet/pods/454c8916-e204-493e-9650-88be170d9224/volumes"
Oct 30 20:03:21.615076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.615051    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5a79d6-7154-4e61-a0de-d5886a1e6726" path="/var/lib/kubelet/pods/8f5a79d6-7154-4e61-a0de-d5886a1e6726/volumes"
Oct 30 20:03:21.992827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.992201    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:03:21.992827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:21.992268    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:03:22.016488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:22.011855    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr" event={"ID":"64e26568-997a-4e1b-a9c3-ec1a57085e7f","Type":"ContainerDied","Data":"a38867aee166a3547c458d28f9b2258bae6cc52be559677b248d6f04affb337f"}
Oct 30 20:03:22.016488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:22.011989    4519 scope.go:122] "RemoveContainer" containerID="f7d5e8060e4fc9ca0dfa563fe49fccbce188bfe06e212fdce8f9338004c21d5f"
Oct 30 20:03:22.016488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:22.012175    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"
Oct 30 20:03:22.074716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:22.074670    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"]
Oct 30 20:03:22.084911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:22.084873    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-55scr"]
Oct 30 20:03:23.614251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:23.614216    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" path="/var/lib/kubelet/pods/64e26568-997a-4e1b-a9c3-ec1a57085e7f/volumes"
Oct 30 20:03:25.599456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.599415    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"]
Oct 30 20:03:25.600212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600194    4519 cpu_manager.go:414] "containerMap: removing container" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy"
Oct 30 20:03:25.600260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600213    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy"
Oct 30 20:03:25.600260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600227    4519 cpu_manager.go:414] "containerMap: removing container" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy"
Oct 30 20:03:25.600260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600232    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy"
Oct 30 20:03:25.600260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600242    4519 cpu_manager.go:414] "containerMap: removing container" podUID="10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" containerName="revoke-keys"
Oct 30 20:03:25.600260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600248    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" containerName="revoke-keys"
Oct 30 20:03:25.600260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600255    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8f5a79d6-7154-4e61-a0de-d5886a1e6726" containerName="discovery"
Oct 30 20:03:25.600260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600260    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5a79d6-7154-4e61-a0de-d5886a1e6726" containerName="discovery"
Oct 30 20:03:25.600508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600362    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="454c8916-e204-493e-9650-88be170d9224" containerName="istio-proxy"
Oct 30 20:03:25.600508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600370    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e26568-997a-4e1b-a9c3-ec1a57085e7f" containerName="istio-proxy"
Oct 30 20:03:25.600508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600378    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" containerName="revoke-keys"
Oct 30 20:03:25.600508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.600384    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5a79d6-7154-4e61-a0de-d5886a1e6726" containerName="discovery"
Oct 30 20:03:25.621494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.611845    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.639667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.639636    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"]
Oct 30 20:03:25.755487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.755427    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.755487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.755643    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.756032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.755803    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.756032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.755855    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftzh\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-kube-api-access-dftzh\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.756032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.756003    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bf24a2-1067-48c5-8219-73b49676fa61-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.756187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.756043    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.756187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.756113    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.756292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.756219    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.756292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.756273    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bf24a2-1067-48c5-8219-73b49676fa61-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.858649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858542    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.858649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858643    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.858882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858749    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.858882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858781    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftzh\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-kube-api-access-dftzh\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.858882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858853    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bf24a2-1067-48c5-8219-73b49676fa61-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858884    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858930    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.859016    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.858939    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.859084    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.859111    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bf24a2-1067-48c5-8219-73b49676fa61-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.859235    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.859462    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.859715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.859702    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bf24a2-1067-48c5-8219-73b49676fa61-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.861721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.861696    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.861821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.861745    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bf24a2-1067-48c5-8219-73b49676fa61-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.878649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.878617    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.878827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.878796    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftzh\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-kube-api-access-dftzh\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:25.940777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:25.940731    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:26.289263 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:03:26.289234    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bf24a2_1067_48c5_8219_73b49676fa61.slice/crio-71f69c32e823180130299efd263b2713ccbd032ea2662ae2b6c2029753c7ca36 WatchSource:0}: Error finding container 71f69c32e823180130299efd263b2713ccbd032ea2662ae2b6c2029753c7ca36: Status 404 returned error can't find the container with id 71f69c32e823180130299efd263b2713ccbd032ea2662ae2b6c2029753c7ca36
Oct 30 20:03:26.291543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:26.291516    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"]
Oct 30 20:03:26.293264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:26.293215    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:26.293365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:26.293304    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:26.293365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:26.293347    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:27.059933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:27.059892    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" event={"ID":"65bf24a2-1067-48c5-8219-73b49676fa61","Type":"ContainerStarted","Data":"0d567a397ba7058f0c90e11592c7b2a3a3d65f775a755e8461ba87f43e9eab0a"}
Oct 30 20:03:27.060574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:27.059964    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" event={"ID":"65bf24a2-1067-48c5-8219-73b49676fa61","Type":"ContainerStarted","Data":"71f69c32e823180130299efd263b2713ccbd032ea2662ae2b6c2029753c7ca36"}
Oct 30 20:03:27.130953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:27.130893    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podStartSLOduration=2.130877304 podStartE2EDuration="2.130877304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:27.115601636 +0000 UTC m=+1731.616305360" watchObservedRunningTime="2026-10-30 20:03:27.130877304 +0000 UTC m=+1731.631581027"
Oct 30 20:03:27.941279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:27.941244    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:27.942733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:27.942710    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:03:27.942825 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:27.942797    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 20:03:28.941227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:28.941187    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:03:28.941692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:28.941257    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 20:03:29.941647 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:29.941604    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:03:29.942092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:29.941670    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 20:03:30.941308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:30.941270    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:03:30.941522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:30.941336    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 20:03:31.942044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:31.942007    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:03:31.942454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:31.942079    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 20:03:32.222323 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:03:32.222236    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:03:32.942115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:32.942077    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:03:32.942587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:32.942137    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 20:03:33.959529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:33.959491    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:33.960742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:33.960678    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:33.962235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:33.962196    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:03:35.793323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.793274    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"]
Oct 30 20:03:35.801185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.801153    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.818858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.818824    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"]
Oct 30 20:03:35.824147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.824117    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ef132f\"/\"openshift-service-ca.crt\""
Oct 30 20:03:35.824147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.824117    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ef132f\"/\"default-dockercfg-m8xfd\""
Oct 30 20:03:35.828325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.828301    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ef132f\"/\"test-model-ef132f-kserve-self-signed-certs\""
Oct 30 20:03:35.828592 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.828574    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ef132f\"/\"kube-root-ca.crt\""
Oct 30 20:03:35.846646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.846609    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-home\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.847104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.847069    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-model-cache\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.847235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.847171    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-dshm\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.847340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.847319    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tmp-dir\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.847520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.847499    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tls-certs\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.847783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.847762    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2gw\" (UniqueName: \"kubernetes.io/projected/cbcb333b-2caf-4b6c-878d-dd782c5a0210-kube-api-access-4t2gw\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.949755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.949708    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-home\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.950577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.950383    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-home\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.950889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.950867    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-model-cache\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.951436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.951417    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-dshm\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.952086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.952066    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tmp-dir\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.952683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.952662    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tls-certs\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.953243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.953224    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2gw\" (UniqueName: \"kubernetes.io/projected/cbcb333b-2caf-4b6c-878d-dd782c5a0210-kube-api-access-4t2gw\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.953860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.952452    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tmp-dir\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.954000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.951315    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-model-cache\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.974976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.974945    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2gw\" (UniqueName: \"kubernetes.io/projected/cbcb333b-2caf-4b6c-878d-dd782c5a0210-kube-api-access-4t2gw\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.985103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.985032    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-dshm\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:35.988091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:35.988027    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tls-certs\") pod \"test-model-ef132f-kserve-68f9588bd9-vzxz9\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") " pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:36.133134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:36.128965    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:36.727511 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:03:36.727424    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbcb333b_2caf_4b6c_878d_dd782c5a0210.slice/crio-c5bba99c82937b2c740b9e1d44bd50da0143c354093270a6d552e94185395309 WatchSource:0}: Error finding container c5bba99c82937b2c740b9e1d44bd50da0143c354093270a6d552e94185395309: Status 404 returned error can't find the container with id c5bba99c82937b2c740b9e1d44bd50da0143c354093270a6d552e94185395309
Oct 30 20:03:36.730819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:36.730790    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"]
Oct 30 20:03:37.306349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:37.306310    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" event={"ID":"cbcb333b-2caf-4b6c-878d-dd782c5a0210","Type":"ContainerStarted","Data":"69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6"}
Oct 30 20:03:37.306885 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:37.306363    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" event={"ID":"cbcb333b-2caf-4b6c-878d-dd782c5a0210","Type":"ContainerStarted","Data":"c5bba99c82937b2c740b9e1d44bd50da0143c354093270a6d552e94185395309"}
Oct 30 20:03:37.347151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:37.347088    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" podStartSLOduration=2.347070587 podStartE2EDuration="2.347070587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:37.346105869 +0000 UTC m=+1741.846809595" watchObservedRunningTime="2026-10-30 20:03:37.347070587 +0000 UTC m=+1741.847774367"
Oct 30 20:03:38.142522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:38.142444    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:40.715031 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.714987    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"]
Oct 30 20:03:40.720686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.720652    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.760975 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.760934    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"]
Oct 30 20:03:40.778554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.778514    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.778772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.778585    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a897e1-923e-465a-ace4-96ffa007311c-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.778772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.778703    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.778772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.778737    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a897e1-923e-465a-ace4-96ffa007311c-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.778923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.778897    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvj7b\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-kube-api-access-dvj7b\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.778963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.778928    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.779100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.779087    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.779139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.779130    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.779287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.779268    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.903641    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.903797    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.903854    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a897e1-923e-465a-ace4-96ffa007311c-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.903902    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.903931    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a897e1-923e-465a-ace4-96ffa007311c-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.904020    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvj7b\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-kube-api-access-dvj7b\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.904057    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.904207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.910558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.904255    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.935097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.931186    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.935097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.931567    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.935097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.932548    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a897e1-923e-465a-ace4-96ffa007311c-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.935097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.932793    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.935097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.933022    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.943311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.943279    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.957490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.944518    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a897e1-923e-465a-ace4-96ffa007311c-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.969326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.969248    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvj7b\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-kube-api-access-dvj7b\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:40.970015 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:40.969991    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:41.045389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:41.045096    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:41.543792 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:03:41.543749    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a897e1_923e_465a_ace4_96ffa007311c.slice/crio-858b957af9cc4b878100cb2286c0533a8750eb5fd2fc3c2a01bfb7cc15c2a1bf WatchSource:0}: Error finding container 858b957af9cc4b878100cb2286c0533a8750eb5fd2fc3c2a01bfb7cc15c2a1bf: Status 404 returned error can't find the container with id 858b957af9cc4b878100cb2286c0533a8750eb5fd2fc3c2a01bfb7cc15c2a1bf
Oct 30 20:03:41.543930 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:41.543842    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"]
Oct 30 20:03:41.547051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:41.546993    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:41.547222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:41.547174    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:41.547222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:41.547211    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:41.899312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:41.899219    4519 scope.go:122] "RemoveContainer" containerID="6026b2c6bcbe73ccbbf6f27a9578b4cddacad7fe93eb5da24a20fab39c2feddd"
Oct 30 20:03:42.508146 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:42.508099    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2" event={"ID":"d0a897e1-923e-465a-ace4-96ffa007311c","Type":"ContainerStarted","Data":"62815355e094e1c2af769492d99a53263886fc03e7d31463b64688f980683565"}
Oct 30 20:03:42.508345 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:42.508176    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2" event={"ID":"d0a897e1-923e-465a-ace4-96ffa007311c","Type":"ContainerStarted","Data":"858b957af9cc4b878100cb2286c0533a8750eb5fd2fc3c2a01bfb7cc15c2a1bf"}
Oct 30 20:03:43.051492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:43.050877    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:43.098490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:43.087459    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:43.172566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:43.167395    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2" podStartSLOduration=3.167369565 podStartE2EDuration="3.167369565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:42.591378402 +0000 UTC m=+1747.092082128" watchObservedRunningTime="2026-10-30 20:03:43.167369565 +0000 UTC m=+1747.668073290"
Oct 30 20:03:43.538491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:43.538000    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:43.572489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:43.571491    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:03:46.143486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.134146    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:46.241486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.240900    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:46.466784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.466716    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"]
Oct 30 20:03:46.505030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.504996    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.512640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.512395    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"]
Oct 30 20:03:46.664487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.641970    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-58d72c-openshift-default-dockercfg-hz76c\""
Oct 30 20:03:46.865719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.765180    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-istio-token\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.894443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.894404    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-socket\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.917704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.917664    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db80b810-4764-4663-839e-1bb464bafcf8-istio-podinfo\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.918173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.918152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-data\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.918357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.918340    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82sg\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-kube-api-access-s82sg\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.918517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.918500    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-envoy\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.918688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.918672    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db80b810-4764-4663-839e-1bb464bafcf8-istiod-ca-cert\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.918809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.918794    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-credential-socket\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:46.918909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:46.918896    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-certs\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.020901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.020823    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db80b810-4764-4663-839e-1bb464bafcf8-istiod-ca-cert\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.022098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.022073    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-credential-socket\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.040493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.022663    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-certs\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.044632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.044594    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-istio-token\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.045220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.045196    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-socket\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.047989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.047963    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db80b810-4764-4663-839e-1bb464bafcf8-istio-podinfo\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.058567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.048804    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-data\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.095707 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.095668    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82sg\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-kube-api-access-s82sg\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.095922 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.095744    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-envoy\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.095987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.043438    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-certs\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.096043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.050333    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-data\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.096043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.022614    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-credential-socket\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.096043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.021976    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db80b810-4764-4663-839e-1bb464bafcf8-istiod-ca-cert\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.096295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.045687    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-socket\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.115395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.115358    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-istio-token\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.120621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.116076    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db80b810-4764-4663-839e-1bb464bafcf8-istio-podinfo\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.179493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.173341    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-envoy\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.179493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.174063    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82sg\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-kube-api-access-s82sg\") pod \"e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") " pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.238487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.235433    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:03:47.426815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.426782    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:47.811828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.811648    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"]
Oct 30 20:03:47.815599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.814554    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:47.815599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.814635    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:47.815599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.814665    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:47.905634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:47.905587    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" event={"ID":"db80b810-4764-4663-839e-1bb464bafcf8","Type":"ContainerStarted","Data":"719fc88beb45ece155833256808452148ef8f2b6546c5c4fdd460fcbf053b91a"}
Oct 30 20:03:48.921014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:48.920901    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" event={"ID":"db80b810-4764-4663-839e-1bb464bafcf8","Type":"ContainerStarted","Data":"321e539d07349d1f39cb41042023fe2899ce5778dae3df5c99e02132eb2c47d2"}
Oct 30 20:03:48.991541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:48.991443    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podStartSLOduration=2.991426353 podStartE2EDuration="2.991426353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:48.975971287 +0000 UTC m=+1753.476675013" watchObservedRunningTime="2026-10-30 20:03:48.991426353 +0000 UTC m=+1753.492130099"
Oct 30 20:03:49.427321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:49.427276    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:49.434061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:49.434033    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:49.932281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:49.932244    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:49.933337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:49.933311    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:03:51.030759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:51.030719    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"]
Oct 30 20:03:51.045632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:51.045591    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3304bad-5vw45"]
Oct 30 20:03:51.618451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:51.618416    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e577efa4-43e6-420d-852a-f003d06b4737" path="/var/lib/kubelet/pods/e577efa4-43e6-420d-852a-f003d06b4737/volumes"
Oct 30 20:03:53.018393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:53.018295    4519 generic.go:356] "Generic (PLEG): container finished" podID="d64c028c-e19b-4191-b338-9b863873c47b" containerID="f8a69c3aa3a260f493591da645463e2fe8252be6d12a248cdefb5dd5c74ed391" exitCode=0
Oct 30 20:03:53.018393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:53.018376    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" event={"ID":"d64c028c-e19b-4191-b338-9b863873c47b","Type":"ContainerDied","Data":"f8a69c3aa3a260f493591da645463e2fe8252be6d12a248cdefb5dd5c74ed391"}
Oct 30 20:03:53.019026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:53.018940    4519 scope.go:122] "RemoveContainer" containerID="f8a69c3aa3a260f493591da645463e2fe8252be6d12a248cdefb5dd5c74ed391"
Oct 30 20:03:53.021168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:53.021140    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:53.021263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:53.021228    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:53.021263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:53.021260    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:54.035413 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:54.035370    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" event={"ID":"d64c028c-e19b-4191-b338-9b863873c47b","Type":"ContainerStarted","Data":"e1dd10253e2cf36c600464a6ddccada56fcab741c4f4b75510866dd0c7088933"}
Oct 30 20:03:54.281849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:54.281810    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:03:54.283080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:54.283046    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:03:54.283265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:54.283105    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:03:55.297026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:55.296986    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:03:55.306083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:55.305650    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:03:55.847944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:55.847889    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"]
Oct 30 20:03:55.877489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:55.871547    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"]
Oct 30 20:03:55.877489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:55.871730    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.039756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.039693    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.040050 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.040027    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.040173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.040158    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.040350 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.040335    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.040506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.040486    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.046719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.046677    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssbct\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-kube-api-access-ssbct\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.047128 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.047104    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.047264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.047248    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.047742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.047713    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5389d26-808a-49fc-8ddb-c349a82fbd17-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.168829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.168787    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.169593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.169566    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.170177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.170153    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssbct\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-kube-api-access-ssbct\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.170774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.170754    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.171310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.171284    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.171908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.171215    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.172263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.172242    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5389d26-808a-49fc-8ddb-c349a82fbd17-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.173221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.173095    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5389d26-808a-49fc-8ddb-c349a82fbd17-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.173711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.173687    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.173861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.173396    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.174017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.174001    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.174454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.174429    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.174949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.174838    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.175137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.174365    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.216166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.216089    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.223244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.223168    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssbct\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-kube-api-access-ssbct\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.234720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.234677    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.237990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.237923    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:56.301123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.300609    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:03:56.301123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.300678    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:03:56.532190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.532088    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:57.033492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:56.997574    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:03:57.044089 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:03:57.044056    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5389d26_808a_49fc_8ddb_c349a82fbd17.slice/crio-a495846fee796253fb920642dd1545978835823de01f1ae20edf6c0869afeb4c WatchSource:0}: Error finding container a495846fee796253fb920642dd1545978835823de01f1ae20edf6c0869afeb4c: Status 404 returned error can't find the container with id a495846fee796253fb920642dd1545978835823de01f1ae20edf6c0869afeb4c
Oct 30 20:03:57.081504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:57.079587    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:57.081504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:57.079703    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:57.081504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:57.079752    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:57.128886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:57.127446    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"]
Oct 30 20:03:57.215744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:57.215676    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" event={"ID":"c5389d26-808a-49fc-8ddb-c349a82fbd17","Type":"ContainerStarted","Data":"a495846fee796253fb920642dd1545978835823de01f1ae20edf6c0869afeb4c"}
Oct 30 20:03:57.315327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:57.315230    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:03:57.315944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:57.315557    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:03:58.284629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:58.284129    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:03:58.284629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:58.284193    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:03:58.361493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:58.352377    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" event={"ID":"c5389d26-808a-49fc-8ddb-c349a82fbd17","Type":"ContainerStarted","Data":"9d2bc017916ae10fd11a404491e677fb222eb9407ccdf6e10837d14fea1aa342"}
Oct 30 20:03:58.515519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:58.511184    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" podStartSLOduration=3.511160919 podStartE2EDuration="3.511160919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:58.487155043 +0000 UTC m=+1762.987858764" watchObservedRunningTime="2026-10-30 20:03:58.511160919 +0000 UTC m=+1763.011864644"
Oct 30 20:03:58.532461 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:58.532394    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:58.533379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:58.533337    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 20:03:58.533537 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:58.533385    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" podUID="c5389d26-808a-49fc-8ddb-c349a82fbd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 20:03:59.282486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:59.281936    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:03:59.282486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:59.282057    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:03:59.821524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:59.815858    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:03:59.821524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:59.815922    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"]
Oct 30 20:03:59.821524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:03:59.816134    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy" containerID="cri-o://bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934" gracePeriod=30
Oct 30 20:04:00.289808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:00.289760    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:04:00.290824 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:00.290778    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:04:00.600756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:00.600638    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:04:00.642493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:00.638537    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"]
Oct 30 20:04:00.799507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:00.782347    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:00.967494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:00.950398    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6cd2ad-openshift-default-dockercfg-r568l\""
Oct 30 20:04:00.989059 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:00.969434    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.003699    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66sxm\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-kube-api-access-66sxm\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.003771    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.003811    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.004074    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb1f85f-6380-4563-88e9-b55d8cb73967-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.004110    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.004178    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.004233    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.009501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.004312    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.105489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.103098    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:04:01.179132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.108172    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.274112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.274010    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66sxm\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-kube-api-access-66sxm\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.332041 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.331999    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.339520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.339478    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.343521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.340065    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb1f85f-6380-4563-88e9-b55d8cb73967-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.371548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.371509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.376152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.376101    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.376511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.376491    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.376625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.112966    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.376691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.368666    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.376691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.129065    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:01.386636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.347536    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.386636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.363531    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb1f85f-6380-4563-88e9-b55d8cb73967-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.387429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.387396    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.440813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.388090    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.441510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.389379    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"]
Oct 30 20:04:01.470659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.387451    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.495616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.485633    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.532692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.532606    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66sxm\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-kube-api-access-66sxm\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.534375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.534288    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 20:04:01.534375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.534366    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 20:04:01.545732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.545658    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:01.709338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:01.709295    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:02.236353 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:04:02.236303    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:04:02.297014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.296973    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:04:02.303581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.303539    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:04:02.396246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.396203    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"]
Oct 30 20:04:02.402524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.402428    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:02.402740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.402542    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:02.402740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.402589    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:02.645827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.645773    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" event={"ID":"6bb1f85f-6380-4563-88e9-b55d8cb73967","Type":"ContainerStarted","Data":"8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa"}
Oct 30 20:04:02.646058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.645858    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" event={"ID":"6bb1f85f-6380-4563-88e9-b55d8cb73967","Type":"ContainerStarted","Data":"2f540d8c6968eacf7f77029ab3b1181ec155f379d68ba50f2781c78acdf05d4e"}
Oct 30 20:04:02.710303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.710221    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podStartSLOduration=2.710192304 podStartE2EDuration="2.710192304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:02.702928683 +0000 UTC m=+1767.203632409" watchObservedRunningTime="2026-10-30 20:04:02.710192304 +0000 UTC m=+1767.210896032"
Oct 30 20:04:02.710699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:02.710345    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:03.718365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:03.718319    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:04.665802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:04.665757    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:04.667196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:04.667147    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:04:05.545262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.544970    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 20:04:05.663002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.662955    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-data\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-certs\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663123    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-podinfo\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663307    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-token\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-token\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663381    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-socket\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663561    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-credential-socket\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663594    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-kube-api-access-5l75s\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-kube-api-access-5l75s\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663646    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-envoy\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.663927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.663767    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istiod-ca-cert\") pod \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\" (UID: \"53cdc4a2-8d8c-4190-9e6e-045499cc99f4\") "
Oct 30 20:04:05.664067 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.664036    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-socket" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:05.664426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.664401    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-certs" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:05.664990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.664965    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.666221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.666202    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.666338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.665241    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-credential-socket" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:05.666425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.665895    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-data" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:05.667047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.667009    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istiod-ca-cert" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:05.669854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.669816    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-token" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:05.675060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.675014    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-kube-api-access-5l75s" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "kube-api-access-5l75s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:05.679057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.678915    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-envoy" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:05.686874 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.686824    4519 generic.go:356] "Generic (PLEG): container finished" podID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerID="bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934" exitCode=0
Oct 30 20:04:05.688496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.687303    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"
Oct 30 20:04:05.688496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.687352    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" event={"ID":"53cdc4a2-8d8c-4190-9e6e-045499cc99f4","Type":"ContainerDied","Data":"bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934"}
Oct 30 20:04:05.688496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.687400    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt" event={"ID":"53cdc4a2-8d8c-4190-9e6e-045499cc99f4","Type":"ContainerDied","Data":"a8a92faa797959d68809bb9c7d2021d7a32fa19a0b7be5c67af9eef8d75dcfed"}
Oct 30 20:04:05.688496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.688233    4519 scope.go:122] "RemoveContainer" containerID="bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934"
Oct 30 20:04:05.696228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.696182    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-podinfo" pod "53cdc4a2-8d8c-4190-9e6e-045499cc99f4" (UID: "53cdc4a2-8d8c-4190-9e6e-045499cc99f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:05.744951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.742681    4519 scope.go:122] "RemoveContainer" containerID="bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934"
Oct 30 20:04:05.748724 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:04:05.745428    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934\": container with ID starting with bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934 not found: ID does not exist" containerID="bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934"
Oct 30 20:04:05.748724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.748651    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934"} err="failed to get container status \"bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934\": rpc error: code = NotFound desc = could not find container \"bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934\": container with ID starting with bc6b07fdd21d4ed7cebbb664a91295387e68af2400f7f5dcfb6fe70be52fe934 not found: ID does not exist"
Oct 30 20:04:05.815193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.800365    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.857076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.857046    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.857337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.857320    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.882493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.857429    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.882865 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.882835    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.914210 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.914112    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l75s\" (UniqueName: \"kubernetes.io/projected/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-kube-api-access-5l75s\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:05.914456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:05.914437    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cdc4a2-8d8c-4190-9e6e-045499cc99f4-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:06.367336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:06.367256    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"]
Oct 30 20:04:06.373767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:06.373699    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-zq4xt"]
Oct 30 20:04:07.821987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:07.788483    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" path="/var/lib/kubelet/pods/53cdc4a2-8d8c-4190-9e6e-045499cc99f4/volumes"
Oct 30 20:04:09.716902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.716854    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"]
Oct 30 20:04:09.717747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.717722    4519 cpu_manager.go:414] "containerMap: removing container" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy"
Oct 30 20:04:09.717874 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.717749    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy"
Oct 30 20:04:09.717968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.717961    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cdc4a2-8d8c-4190-9e6e-045499cc99f4" containerName="istio-proxy"
Oct 30 20:04:09.722097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.722066    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.745694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.745661    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-936f70\"/\"openshift-service-ca.crt\""
Oct 30 20:04:09.746059 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.745661    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-936f70\"/\"kube-root-ca.crt\""
Oct 30 20:04:09.773788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.773749    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"]
Oct 30 20:04:09.810903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.810773    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-936f70\"/\"default-dockercfg-gv8bw\""
Oct 30 20:04:09.810903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.810735    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-936f70\"/\"test-model-936f70-kserve-self-signed-certs\""
Oct 30 20:04:09.822382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.822338    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf82f\" (UniqueName: \"kubernetes.io/projected/9de8bd51-bab6-409e-aa03-9a40668303a9-kube-api-access-rf82f\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.822591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.822539    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de8bd51-bab6-409e-aa03-9a40668303a9-tls-certs\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.822640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.822593    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-model-cache\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.823030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.823004    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-tmp-dir\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.823207 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.823185    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-home\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.823392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.823369    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-dshm\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.924695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.924655    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-home\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.924903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.924759    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-dshm\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.924903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.924834    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf82f\" (UniqueName: \"kubernetes.io/projected/9de8bd51-bab6-409e-aa03-9a40668303a9-kube-api-access-rf82f\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.924997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.924952    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de8bd51-bab6-409e-aa03-9a40668303a9-tls-certs\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.924997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.924987    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-model-cache\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.925226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.925207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-tmp-dir\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.925266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.925243    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-home\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.926093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.926068    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-model-cache\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.926346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.926319    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-tmp-dir\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.929757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.929724    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-dshm\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.930717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.930685    4519 generic.go:356] "Generic (PLEG): container finished" podID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerID="d8deb4449947ace196ed08c659ba81f71f59e3414568f8e277dcb10aae0c3cd4" exitCode=0
Oct 30 20:04:09.930861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.930758    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" event={"ID":"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8","Type":"ContainerDied","Data":"d8deb4449947ace196ed08c659ba81f71f59e3414568f8e277dcb10aae0c3cd4"}
Oct 30 20:04:09.931376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.931354    4519 scope.go:122] "RemoveContainer" containerID="d8deb4449947ace196ed08c659ba81f71f59e3414568f8e277dcb10aae0c3cd4"
Oct 30 20:04:09.934168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.934142    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de8bd51-bab6-409e-aa03-9a40668303a9-tls-certs\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:09.934271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.934189    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:09.934341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.934321    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:09.934398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.934382    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:09.945929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:09.945842    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf82f\" (UniqueName: \"kubernetes.io/projected/9de8bd51-bab6-409e-aa03-9a40668303a9-kube-api-access-rf82f\") pod \"test-model-936f70-kserve-85fc5647d5-69cwh\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") " pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:10.077633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:10.075457    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:10.944054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:10.944007    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" event={"ID":"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8","Type":"ContainerStarted","Data":"dfc5c9369c8dcc2c783c11eaff4b68805e0e3bb671345a77d7cf6f17e8ba967d"}
Oct 30 20:04:10.986446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:10.986408    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"]
Oct 30 20:04:11.866492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:11.860905    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:04:11.866492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:11.861120    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:04:11.866492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:11.861178    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:04:11.959101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:11.958999    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:04:11.990431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:11.990387    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh" event={"ID":"9de8bd51-bab6-409e-aa03-9a40668303a9","Type":"ContainerStarted","Data":"1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e"}
Oct 30 20:04:12.011877 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:12.011834    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh" event={"ID":"9de8bd51-bab6-409e-aa03-9a40668303a9","Type":"ContainerStarted","Data":"29f8b02b5b927335dd8b5990a4b6d65840ba254f907832a297002f76db9c3a9c"}
Oct 30 20:04:12.055435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:12.055360    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh" podStartSLOduration=3.055337136 podStartE2EDuration="3.055337136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:12.040957864 +0000 UTC m=+1776.541661589" watchObservedRunningTime="2026-10-30 20:04:12.055337136 +0000 UTC m=+1776.556040860"
Oct 30 20:04:12.078110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:12.078070    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:12.871549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:12.868751    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:04:12.871549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:12.868852    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:04:13.866494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:13.865577    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:04:13.866494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:13.865646    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:04:14.866493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:14.864921    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:04:14.866493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:14.865044    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:04:14.947495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:14.942626    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"]
Oct 30 20:04:14.947495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:14.943064    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" podUID="c5389d26-808a-49fc-8ddb-c349a82fbd17" containerName="istio-proxy" containerID="cri-o://9d2bc017916ae10fd11a404491e677fb222eb9407ccdf6e10837d14fea1aa342" gracePeriod=30
Oct 30 20:04:15.013950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.013909    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"]
Oct 30 20:04:15.014625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.014561    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" containerID="cri-o://b6f382f3e9078a5ca769d79c02de50ec0d5dae90adb60444c76a0e3d7ef725db" gracePeriod=30
Oct 30 20:04:15.023865 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.023831    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"]
Oct 30 20:04:15.024089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.024045    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2" podUID="d0a897e1-923e-465a-ace4-96ffa007311c" containerName="istio-proxy" containerID="cri-o://62815355e094e1c2af769492d99a53263886fc03e7d31463b64688f980683565" gracePeriod=30
Oct 30 20:04:15.028966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.028930    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"]
Oct 30 20:04:15.029201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.029179    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" containerID="cri-o://0d567a397ba7058f0c90e11592c7b2a3a3d65f775a755e8461ba87f43e9eab0a" gracePeriod=30
Oct 30 20:04:15.599669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.599161    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:04:15.599669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.599246    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" podUID="c5389d26-808a-49fc-8ddb-c349a82fbd17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:04:15.898449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.898336    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:04:15.898449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:15.898411    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:04:16.882587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:16.881661    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:04:16.882587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:16.881733    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:04:17.664286 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:17.664245    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:04:17.665213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:17.664906    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:04:17.865599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:17.865561    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:04:17.865888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:17.865860    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 20:04:18.866289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:18.866246    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:04:18.867741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:18.867717    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:04:18.964413 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:18.964356    4519 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 20:04:18.964671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:18.964427    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:04:20.096002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.095732    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:20.126341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.119543    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:20.337854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.337393    4519 generic.go:356] "Generic (PLEG): container finished" podID="65bf24a2-1067-48c5-8219-73b49676fa61" containerID="0d567a397ba7058f0c90e11592c7b2a3a3d65f775a755e8461ba87f43e9eab0a" exitCode=0
Oct 30 20:04:20.337854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.337596    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" event={"ID":"65bf24a2-1067-48c5-8219-73b49676fa61","Type":"ContainerDied","Data":"0d567a397ba7058f0c90e11592c7b2a3a3d65f775a755e8461ba87f43e9eab0a"}
Oct 30 20:04:20.339872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.339631    4519 generic.go:356] "Generic (PLEG): container finished" podID="c5389d26-808a-49fc-8ddb-c349a82fbd17" containerID="9d2bc017916ae10fd11a404491e677fb222eb9407ccdf6e10837d14fea1aa342" exitCode=0
Oct 30 20:04:20.340068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.339917    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" event={"ID":"c5389d26-808a-49fc-8ddb-c349a82fbd17","Type":"ContainerDied","Data":"9d2bc017916ae10fd11a404491e677fb222eb9407ccdf6e10837d14fea1aa342"}
Oct 30 20:04:20.344497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.342277    4519 generic.go:356] "Generic (PLEG): container finished" podID="d0a897e1-923e-465a-ace4-96ffa007311c" containerID="62815355e094e1c2af769492d99a53263886fc03e7d31463b64688f980683565" exitCode=0
Oct 30 20:04:20.344497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.342339    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2" event={"ID":"d0a897e1-923e-465a-ace4-96ffa007311c","Type":"ContainerDied","Data":"62815355e094e1c2af769492d99a53263886fc03e7d31463b64688f980683565"}
Oct 30 20:04:20.350012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.345351    4519 generic.go:356] "Generic (PLEG): container finished" podID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerID="b6f382f3e9078a5ca769d79c02de50ec0d5dae90adb60444c76a0e3d7ef725db" exitCode=0
Oct 30 20:04:20.350012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.345423    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" event={"ID":"40ba3d21-8105-42b2-bc93-abfb6953efca","Type":"ContainerDied","Data":"b6f382f3e9078a5ca769d79c02de50ec0d5dae90adb60444c76a0e3d7ef725db"}
Oct 30 20:04:20.350012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.345529    4519 scope.go:122] "RemoveContainer" containerID="2256ce950e5c68549704876d5f94a28c3b7c26a010c3899121a4d32118a3b60f"
Oct 30 20:04:20.350012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.367737    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:04:20.437629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.437254    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:04:20.505493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.500924    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.531790    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-certs\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.531880    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-credential-socket\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.531938    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-istio-token\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-istio-token\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.531972    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65bf24a2-1067-48c5-8219-73b49676fa61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bf24a2-1067-48c5-8219-73b49676fa61-istio-podinfo\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532038    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-kube-api-access-ssbct\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-kube-api-access-ssbct\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532157    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-socket\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532206    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-kube-api-access-dftzh\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-kube-api-access-dftzh\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532252    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-socket\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532322    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5389d26-808a-49fc-8ddb-c349a82fbd17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5389d26-808a-49fc-8ddb-c349a82fbd17-istiod-ca-cert\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532359    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-podinfo\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532438    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-envoy\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532503    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-credential-socket\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532630    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-envoy\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-data\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532713    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-data\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532852    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-token\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-token\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532912    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bf24a2-1067-48c5-8219-73b49676fa61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bf24a2-1067-48c5-8219-73b49676fa61-istiod-ca-cert\") pod \"65bf24a2-1067-48c5-8219-73b49676fa61\" (UID: \"65bf24a2-1067-48c5-8219-73b49676fa61\") "
Oct 30 20:04:20.534488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.532991    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-certs\") pod \"c5389d26-808a-49fc-8ddb-c349a82fbd17\" (UID: \"c5389d26-808a-49fc-8ddb-c349a82fbd17\") "
Oct 30 20:04:20.535618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.534779    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-certs" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.535618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.535127    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-credential-socket" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.537916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.537057    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5389d26-808a-49fc-8ddb-c349a82fbd17-istiod-ca-cert" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:20.537916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.537300    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-socket" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.537916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.537640    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-socket" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.538551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.538525    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-data" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.538901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.538877    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-credential-socket" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.541191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.541167    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-data" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.541558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.541535    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bf24a2-1067-48c5-8219-73b49676fa61-istiod-ca-cert" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:20.566024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.565966    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-certs" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.592456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.588712    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65bf24a2-1067-48c5-8219-73b49676fa61-istio-podinfo" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:20.592829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.588749    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-envoy" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.593569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.593506    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-podinfo" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:20.594130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.594083    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-token" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.594572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.594537    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-istio-token" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.594720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.594549    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-kube-api-access-dftzh" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "kube-api-access-dftzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.596735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.596696    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-kube-api-access-ssbct" pod "c5389d26-808a-49fc-8ddb-c349a82fbd17" (UID: "c5389d26-808a-49fc-8ddb-c349a82fbd17"). InnerVolumeSpecName "kube-api-access-ssbct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.597194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.597167    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-envoy" pod "65bf24a2-1067-48c5-8219-73b49676fa61" (UID: "65bf24a2-1067-48c5-8219-73b49676fa61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636557    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636595    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dftzh\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-kube-api-access-dftzh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636611    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636626    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5389d26-808a-49fc-8ddb-c349a82fbd17-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636641    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636654    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636666    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636680    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636693    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636706    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636718    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636730    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bf24a2-1067-48c5-8219-73b49676fa61-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636743    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5389d26-808a-49fc-8ddb-c349a82fbd17-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636755    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636768    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bf24a2-1067-48c5-8219-73b49676fa61-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636780    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bf24a2-1067-48c5-8219-73b49676fa61-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636793    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bf24a2-1067-48c5-8219-73b49676fa61-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.636868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.636808    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssbct\" (UniqueName: \"kubernetes.io/projected/c5389d26-808a-49fc-8ddb-c349a82fbd17-kube-api-access-ssbct\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.644926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.644354    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:04:20.647016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.646917    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 20:04:20.737394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737362    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-kube-api-access-dvj7b\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-kube-api-access-dvj7b\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.737394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737398    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-data\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.737729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737423    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-data\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.737729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737536    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-envoy\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.737729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737612    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ba3d21-8105-42b2-bc93-abfb6953efca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba3d21-8105-42b2-bc93-abfb6953efca-istiod-ca-cert\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.737729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-socket\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.737729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-istio-token\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-istio-token\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.738032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737788    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0a897e1-923e-465a-ace4-96ffa007311c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a897e1-923e-465a-ace4-96ffa007311c-istio-podinfo\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.738032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737806    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-data" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.738032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.737946    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-podinfo\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.738032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-certs\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.738242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738069    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-token\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-token\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.738242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738163    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-envoy\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.738242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738195    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a897e1-923e-465a-ace4-96ffa007311c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a897e1-923e-465a-ace4-96ffa007311c-istiod-ca-cert\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.738406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-credential-socket\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.738406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738298    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-socket\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.738406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738322    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-certs\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.738406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738378    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-kube-api-access-bdmzm\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-kube-api-access-bdmzm\") pod \"40ba3d21-8105-42b2-bc93-abfb6953efca\" (UID: \"40ba3d21-8105-42b2-bc93-abfb6953efca\") "
Oct 30 20:04:20.738636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.738440    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-credential-socket\") pod \"d0a897e1-923e-465a-ace4-96ffa007311c\" (UID: \"d0a897e1-923e-465a-ace4-96ffa007311c\") "
Oct 30 20:04:20.739196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.739160    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ba3d21-8105-42b2-bc93-abfb6953efca-istiod-ca-cert" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:20.739338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.739193    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.739449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.739430    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-credential-socket" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.739539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.739460    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-data" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.740693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.740662    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-kube-api-access-dvj7b" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "kube-api-access-dvj7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.740971 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.740944    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-socket" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.741177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.741152    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-token" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.741264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.741178    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a897e1-923e-465a-ace4-96ffa007311c-istiod-ca-cert" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:20.741332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.741317    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-socket" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.741406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.741370    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-credential-socket" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.742116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.741790    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-certs" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.742116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.742024    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-envoy" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.742289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.742128    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-certs" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.742795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.742766    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-envoy" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:20.744026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.743998    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-kube-api-access-bdmzm" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "kube-api-access-bdmzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.744144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.744062    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-podinfo" pod "40ba3d21-8105-42b2-bc93-abfb6953efca" (UID: "40ba3d21-8105-42b2-bc93-abfb6953efca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:20.745188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.745161    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-istio-token" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.746774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.746745    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0a897e1-923e-465a-ace4-96ffa007311c-istio-podinfo" pod "d0a897e1-923e-465a-ace4-96ffa007311c" (UID: "d0a897e1-923e-465a-ace4-96ffa007311c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841206    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a897e1-923e-465a-ace4-96ffa007311c-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841245    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841253    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841262    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841275    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841287    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a897e1-923e-465a-ace4-96ffa007311c-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841297    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841306    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841316    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841325    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmzm\" (UniqueName: \"kubernetes.io/projected/40ba3d21-8105-42b2-bc93-abfb6953efca-kube-api-access-bdmzm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841334    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841346    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvj7b\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-kube-api-access-dvj7b\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841360    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba3d21-8105-42b2-bc93-abfb6953efca-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841369    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841377    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba3d21-8105-42b2-bc93-abfb6953efca-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841385    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a897e1-923e-465a-ace4-96ffa007311c-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.841845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:20.841392    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a897e1-923e-465a-ace4-96ffa007311c-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:21.364451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.364416    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"
Oct 30 20:04:21.365851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.364421    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2" event={"ID":"d0a897e1-923e-465a-ace4-96ffa007311c","Type":"ContainerDied","Data":"858b957af9cc4b878100cb2286c0533a8750eb5fd2fc3c2a01bfb7cc15c2a1bf"}
Oct 30 20:04:21.366007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.365904    4519 scope.go:122] "RemoveContainer" containerID="62815355e094e1c2af769492d99a53263886fc03e7d31463b64688f980683565"
Oct 30 20:04:21.367789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.367756    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"
Oct 30 20:04:21.369581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.367778    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2" event={"ID":"40ba3d21-8105-42b2-bc93-abfb6953efca","Type":"ContainerDied","Data":"39972511aeefcb187ef0f859194f41dd2fe4d51f8718500e0e5b4b060fc05073"}
Oct 30 20:04:21.374870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.374828    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r" event={"ID":"65bf24a2-1067-48c5-8219-73b49676fa61","Type":"ContainerDied","Data":"71f69c32e823180130299efd263b2713ccbd032ea2662ae2b6c2029753c7ca36"}
Oct 30 20:04:21.375131 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.374844    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"
Oct 30 20:04:21.379116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.379088    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"
Oct 30 20:04:21.380781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.380606    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq" event={"ID":"c5389d26-808a-49fc-8ddb-c349a82fbd17","Type":"ContainerDied","Data":"a495846fee796253fb920642dd1545978835823de01f1ae20edf6c0869afeb4c"}
Oct 30 20:04:21.393886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.393861    4519 scope.go:122] "RemoveContainer" containerID="b6f382f3e9078a5ca769d79c02de50ec0d5dae90adb60444c76a0e3d7ef725db"
Oct 30 20:04:21.406589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.406564    4519 scope.go:122] "RemoveContainer" containerID="0d567a397ba7058f0c90e11592c7b2a3a3d65f775a755e8461ba87f43e9eab0a"
Oct 30 20:04:21.419034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.419007    4519 scope.go:122] "RemoveContainer" containerID="9d2bc017916ae10fd11a404491e677fb222eb9407ccdf6e10837d14fea1aa342"
Oct 30 20:04:21.467205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.467164    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"]
Oct 30 20:04:21.480915 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.480864    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-t4tl2"]
Oct 30 20:04:21.521578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.521531    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"]
Oct 30 20:04:21.525843 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.525804    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-v8ls2"]
Oct 30 20:04:21.556733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.556677    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"]
Oct 30 20:04:21.564333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.564287    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-7wb4r"]
Oct 30 20:04:21.594709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.594667    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"]
Oct 30 20:04:21.600985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.600949    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-wdlxq"]
Oct 30 20:04:21.614902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.614802    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" path="/var/lib/kubelet/pods/40ba3d21-8105-42b2-bc93-abfb6953efca/volumes"
Oct 30 20:04:21.615427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.615412    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" path="/var/lib/kubelet/pods/65bf24a2-1067-48c5-8219-73b49676fa61/volumes"
Oct 30 20:04:21.616024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.616004    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5389d26-808a-49fc-8ddb-c349a82fbd17" path="/var/lib/kubelet/pods/c5389d26-808a-49fc-8ddb-c349a82fbd17/volumes"
Oct 30 20:04:21.617000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:21.616982    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a897e1-923e-465a-ace4-96ffa007311c" path="/var/lib/kubelet/pods/d0a897e1-923e-465a-ace4-96ffa007311c/volumes"
Oct 30 20:04:32.251529 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:04:32.250796    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:04:34.582496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:34.575704    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:04:34.582496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:34.575793    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:35.231315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:35.231268    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"]
Oct 30 20:04:35.241927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:35.241865    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-652137-7hxwb"]
Oct 30 20:04:35.616916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:35.616821    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d24f1e-023a-48b4-bcf4-7e1b9a3747a1" path="/var/lib/kubelet/pods/10d24f1e-023a-48b4-bcf4-7e1b9a3747a1/volumes"
Oct 30 20:04:36.395796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.395725    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": read tcp 10.133.0.2:44132->10.133.0.195:15021: read: connection reset by peer" start-of-body=
Oct 30 20:04:36.396032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.395795    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": read tcp 10.133.0.2:44132->10.133.0.195:15021: read: connection reset by peer"
Oct 30 20:04:36.895238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.895195    4519 generic.go:356] "Generic (PLEG): container finished" podID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerID="0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c" exitCode=0
Oct 30 20:04:36.895856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.895266    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" event={"ID":"aa8b2862-9442-47d5-af3c-ff55bb26b17c","Type":"ContainerDied","Data":"0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c"}
Oct 30 20:04:36.895923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.895890    4519 scope.go:122] "RemoveContainer" containerID="0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c"
Oct 30 20:04:36.897625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.897597    4519 generic.go:356] "Generic (PLEG): container finished" podID="f4a28953-5648-4dad-8d3e-fe423819340f" containerID="2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25" exitCode=0
Oct 30 20:04:36.897719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.897666    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" event={"ID":"f4a28953-5648-4dad-8d3e-fe423819340f","Type":"ContainerDied","Data":"2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25"}
Oct 30 20:04:36.898097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.898082    4519 scope.go:122] "RemoveContainer" containerID="2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25"
Oct 30 20:04:36.898143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.898114    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:36.898196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.898183    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:36.898233 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.898224    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:36.900032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.900003    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:36.900125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.900081    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:36.900125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:36.900116    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:37.919875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:37.919825    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" event={"ID":"aa8b2862-9442-47d5-af3c-ff55bb26b17c","Type":"ContainerStarted","Data":"4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae"}
Oct 30 20:04:37.923868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:37.923833    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" event={"ID":"f4a28953-5648-4dad-8d3e-fe423819340f","Type":"ContainerStarted","Data":"4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd"}
Oct 30 20:04:38.390557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:38.386931    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:04:38.390557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:38.387074    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:38.390557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:38.387120    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:38.406614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:38.400553    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:04:38.406614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:38.400720    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:38.406614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:38.400766    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:39.399505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:39.398926    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:39.399505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:39.399002    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:39.432375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:39.404868    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:39.432375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:39.404935    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:40.392238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:40.383976    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:40.392238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:40.384063    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:40.392585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:40.392529    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:40.404501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:40.392706    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:41.391100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:41.384484    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:41.391100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:41.384567    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:41.391100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:41.390885    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:41.391100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:41.390963    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:42.391909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:42.391057    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:42.391909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:42.391137    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:42.391909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:42.391814    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:42.391909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:42.391854    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:42.449899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:42.449863    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"]
Oct 30 20:04:42.450417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:42.450384    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy" containerID="cri-o://377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4" gracePeriod=30
Oct 30 20:04:43.106493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.106163    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:04:43.129490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.129042    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:43.142059 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.132430    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:43.155861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.147792    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:04:43.155861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.148872    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:04:43.155861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.149919    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 20:04:43.385580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.384455    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:43.385580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.384562    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:43.399508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.392645    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:43.399508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:43.392716    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:44.037366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.037332    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"]
Oct 30 20:04:44.039356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.039320    4519 cpu_manager.go:414] "containerMap: removing container" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy"
Oct 30 20:04:44.039608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.039588    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy"
Oct 30 20:04:44.039812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.039796    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c5389d26-808a-49fc-8ddb-c349a82fbd17" containerName="istio-proxy"
Oct 30 20:04:44.050587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.050550    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5389d26-808a-49fc-8ddb-c349a82fbd17" containerName="istio-proxy"
Oct 30 20:04:44.050901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.050885    4519 cpu_manager.go:414] "containerMap: removing container" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy"
Oct 30 20:04:44.051011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.050998    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy"
Oct 30 20:04:44.051104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.051095    4519 cpu_manager.go:414] "containerMap: removing container" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy"
Oct 30 20:04:44.051199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.051187    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy"
Oct 30 20:04:44.051294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.051284    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d0a897e1-923e-465a-ace4-96ffa007311c" containerName="istio-proxy"
Oct 30 20:04:44.051366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.051356    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a897e1-923e-465a-ace4-96ffa007311c" containerName="istio-proxy"
Oct 30 20:04:44.051958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.051930    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy"
Oct 30 20:04:44.066779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.066748    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ba3d21-8105-42b2-bc93-abfb6953efca" containerName="istio-proxy"
Oct 30 20:04:44.066986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.066971    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a897e1-923e-465a-ace4-96ffa007311c" containerName="istio-proxy"
Oct 30 20:04:44.067168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.067155    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bf24a2-1067-48c5-8219-73b49676fa61" containerName="istio-proxy"
Oct 30 20:04:44.067307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.067295    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5389d26-808a-49fc-8ddb-c349a82fbd17" containerName="istio-proxy"
Oct 30 20:04:44.154493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.147984    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.170492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.155761    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"]
Oct 30 20:04:44.171133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.171085    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c8fc40-openshift-default-dockercfg-jszsj\""
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201299    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-token\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201404    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-podinfo\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201436    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-socket\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201508    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-envoy\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201591    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fjk\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-kube-api-access-k2fjk\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201726    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-data\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201751    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istiod-ca-cert\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201904    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-certs\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.210288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.201972    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-credential-socket\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.308336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.306347    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-podinfo\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.316490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.309094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-socket\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.316490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.309191    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-envoy\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.316490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.309291    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fjk\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-kube-api-access-k2fjk\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.362259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.329724    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-socket\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.393667    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.393735    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.366800    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-data\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.397914    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.397982    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.398646    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-podinfo\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.366170    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-data\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.399916    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istiod-ca-cert\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.400138    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-certs\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.400225    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-credential-socket\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.400329    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-token\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.401790    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istiod-ca-cert\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.402093    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-certs\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.406999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.402350    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-credential-socket\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.441975 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.441939    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-token\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.456372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.456290    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fjk\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-kube-api-access-k2fjk\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.456890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.456859    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-envoy\") pod \"e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") " pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.605627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.605517    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:44.901020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:44.899801    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:45.209837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.203381    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"]
Oct 30 20:04:45.210194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.210146    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:45.210402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.210383    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:45.210571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.210549    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:04:45.384028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.383986    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:04:45.384780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.384276    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 20:04:45.391250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.391144    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 20:04:45.391250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.391194    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 20:04:45.431492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.431296    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" event={"ID":"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e","Type":"ContainerStarted","Data":"da9cf93be5dba0da7c8924357effb264738b1470d96d8cc910d42f9bfa31f2e1"}
Oct 30 20:04:45.431492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.431358    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" event={"ID":"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e","Type":"ContainerStarted","Data":"d46a9456aca1d963add695badacf8de63f98aaa2f99a9501740a3fca373fe19f"}
Oct 30 20:04:45.493552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.491440    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podStartSLOduration=2.491416659 podStartE2EDuration="2.491416659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:45.480070284 +0000 UTC m=+1809.980774010" watchObservedRunningTime="2026-10-30 20:04:45.491416659 +0000 UTC m=+1809.992120384"
Oct 30 20:04:45.609512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:45.606610    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:46.388532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:46.388450    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:04:46.389073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:46.389044    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:04:46.390182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:46.390159    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:04:46.397192 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:46.397159    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:04:46.397513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:46.397491    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:04:46.398348 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:46.398330    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:04:46.616587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:46.616552    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:47.448545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:47.448461    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:47.449612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:47.449579    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:04:48.284545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.284517    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:04:48.397934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.397881    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-podinfo\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.398126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398105    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-kube-api-access-wgz6q\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-kube-api-access-wgz6q\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.398253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398237    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istiod-ca-cert\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398365    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-token\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-token\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398413    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-envoy\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398452    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-data\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398543    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-certs\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398580    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-credential-socket\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.398702    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-socket\") pod \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\" (UID: \"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37\") "
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.399305    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-data" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:48.399519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.399333    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-certs" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:48.400096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.399553    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-credential-socket" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:48.400096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.399836    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.400096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.399860    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.400096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.399874    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.400096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.400057    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istiod-ca-cert" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:48.400506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.400438    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-socket" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:48.400918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.400882    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-podinfo" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:48.401089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.401060    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-kube-api-access-wgz6q" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "kube-api-access-wgz6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:48.401365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.401323    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-envoy" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:48.402046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.402014    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-token" pod "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" (UID: "247c9ec7-c64a-4e57-96a1-0a26a7cf1a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:48.458637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.458597    4519 generic.go:356] "Generic (PLEG): container finished" podID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerID="377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4" exitCode=0
Oct 30 20:04:48.458841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.458655    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"
Oct 30 20:04:48.458841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.458689    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" event={"ID":"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37","Type":"ContainerDied","Data":"377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4"}
Oct 30 20:04:48.458841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.458792    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw" event={"ID":"247c9ec7-c64a-4e57-96a1-0a26a7cf1a37","Type":"ContainerDied","Data":"8a5be9bd26f6a2af19fd94b64365f9add5d9038f3a85750c0706b3a020d16d41"}
Oct 30 20:04:48.458841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.458825    4519 scope.go:122] "RemoveContainer" containerID="377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4"
Oct 30 20:04:48.472880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.472849    4519 scope.go:122] "RemoveContainer" containerID="bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f"
Oct 30 20:04:48.484711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.484685    4519 scope.go:122] "RemoveContainer" containerID="377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4"
Oct 30 20:04:48.485062 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:04:48.485041    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4\": container with ID starting with 377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4 not found: ID does not exist" containerID="377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4"
Oct 30 20:04:48.485116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.485075    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4"} err="failed to get container status \"377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4\": rpc error: code = NotFound desc = could not find container \"377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4\": container with ID starting with 377f7f928fc5caf01e7ce29b0c3aac3cc75d1a99cc51ec0a9cefe92ca46770e4 not found: ID does not exist"
Oct 30 20:04:48.485116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.485100    4519 scope.go:122] "RemoveContainer" containerID="bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f"
Oct 30 20:04:48.485387 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:04:48.485364    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f\": container with ID starting with bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f not found: ID does not exist" containerID="bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f"
Oct 30 20:04:48.485460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.485393    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f"} err="failed to get container status \"bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f\": rpc error: code = NotFound desc = could not find container \"bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f\": container with ID starting with bbb3002c3cd8de2d228add4b6425a299a67b1e3804bd8848e399ab9b25a4720f not found: ID does not exist"
Oct 30 20:04:48.501900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.501813    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.501900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.501855    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.501900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.501871    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.501900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.501884    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.501900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.501898    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgz6q\" (UniqueName: \"kubernetes.io/projected/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-kube-api-access-wgz6q\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.502176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.501913    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:04:48.516327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.516288    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"]
Oct 30 20:04:48.523081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:48.523029    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-652137-openshift-default-697fcbcbff-tw6jw"]
Oct 30 20:04:49.614397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:04:49.614356    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" path="/var/lib/kubelet/pods/247c9ec7-c64a-4e57-96a1-0a26a7cf1a37/volumes"
Oct 30 20:05:02.225262 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:05:02.225214    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:05:17.865492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:17.864606    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:18.031544 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:18.031212    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:25.284912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:25.284875    4519 generic.go:356] "Generic (PLEG): container finished" podID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerID="a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3" exitCode=0
Oct 30 20:05:25.285438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:25.284961    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" event={"ID":"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e","Type":"ContainerDied","Data":"a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3"}
Oct 30 20:05:25.285683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:25.285660    4519 scope.go:122] "RemoveContainer" containerID="a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3"
Oct 30 20:05:25.288035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:25.287996    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:25.288152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:25.288076    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:25.288152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:25.288113    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:26.306998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:26.306937    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" event={"ID":"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e","Type":"ContainerStarted","Data":"6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d"}
Oct 30 20:05:26.341493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:26.339991    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:05:26.341493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:26.341406    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:05:26.341797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:26.341563    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 20:05:27.339552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:27.339484    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:05:27.339552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:27.339550    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 20:05:28.339396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:28.339352    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:05:28.339662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:28.339424    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 20:05:29.339896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:29.339855    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:05:29.340344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:29.339920    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 20:05:30.083089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:30.078635    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"]
Oct 30 20:05:30.114492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:30.107483    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f00559-sdv4s"]
Oct 30 20:05:30.206493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:30.202583    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:05:30.349621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:30.348311    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:05:30.349621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:30.348399    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 20:05:31.040502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:31.040384    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"]
Oct 30 20:05:31.107448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:31.107411    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40400581-cs58v"]
Oct 30 20:05:31.339998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:31.339893    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:05:31.340307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:31.340269    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 20:05:31.625504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:31.625389    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec21be5-5c47-4949-82fb-3d2322ac0ec7" path="/var/lib/kubelet/pods/1ec21be5-5c47-4949-82fb-3d2322ac0ec7/volumes"
Oct 30 20:05:31.626096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:31.626073    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c104561-b3ad-4197-b6fd-4227e414f091" path="/var/lib/kubelet/pods/2c104561-b3ad-4197-b6fd-4227e414f091/volumes"
Oct 30 20:05:32.242955 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:05:32.242876    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:05:32.339153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:32.339112    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:05:32.339364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:32.339175    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 20:05:33.369494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:33.364548    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:05:33.369494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:33.368725    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:05:39.944208 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.944162    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"]
Oct 30 20:05:39.945380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.945347    4519 cpu_manager.go:414] "containerMap: removing container" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy"
Oct 30 20:05:39.945493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.945385    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy"
Oct 30 20:05:39.945689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.945668    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy"
Oct 30 20:05:39.945777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.945703    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy"
Oct 30 20:05:39.952236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.952201    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:39.982111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.980977    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7b5635\"/\"openshift-service-ca.crt\""
Oct 30 20:05:39.982111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.981120    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7b5635\"/\"kube-root-ca.crt\""
Oct 30 20:05:39.991436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.991391    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"]
Oct 30 20:05:39.993166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.993134    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7b5635\"/\"default-dockercfg-cv9jr\""
Oct 30 20:05:39.993334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:39.993245    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7b5635\"/\"test-model-7b5635-kserve-self-signed-certs\""
Oct 30 20:05:40.023577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.023277    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a651deb-f472-4c76-aa17-151115629af0-tls-certs\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.023577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.023415    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-home\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.023886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.023658    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-tmp-dir\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.023948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.023894    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-model-cache\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.024000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.023946    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-624hc\" (UniqueName: \"kubernetes.io/projected/9a651deb-f472-4c76-aa17-151115629af0-kube-api-access-624hc\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.024210 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.024027    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-dshm\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.126530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.126143    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-home\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.126530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.126271    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-tmp-dir\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.126530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.126490    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-model-cache\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.126863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.126569    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-624hc\" (UniqueName: \"kubernetes.io/projected/9a651deb-f472-4c76-aa17-151115629af0-kube-api-access-624hc\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.126863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.126645    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-dshm\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.126863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.126703    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a651deb-f472-4c76-aa17-151115629af0-tls-certs\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.126863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.126816    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-home\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.131774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.127362    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-model-cache\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.131774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.127679    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-tmp-dir\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.131774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.131260    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a651deb-f472-4c76-aa17-151115629af0-tls-certs\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.131774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.131525    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-dshm\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.164399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.164357    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-624hc\" (UniqueName: \"kubernetes.io/projected/9a651deb-f472-4c76-aa17-151115629af0-kube-api-access-624hc\") pod \"test-model-7b5635-kserve-599f6865bc-whzt7\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") " pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.299909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.299811    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:40.819790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:40.819707    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"]
Oct 30 20:05:40.825821 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:05:40.821828    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a651deb_f472_4c76_aa17_151115629af0.slice/crio-b91e88269d4f76ff891624ad92de0b37da21ead78b4f0d037a7ce55e927040e8 WatchSource:0}: Error finding container b91e88269d4f76ff891624ad92de0b37da21ead78b4f0d037a7ce55e927040e8: Status 404 returned error can't find the container with id b91e88269d4f76ff891624ad92de0b37da21ead78b4f0d037a7ce55e927040e8
Oct 30 20:05:41.594154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:41.594103    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7" event={"ID":"9a651deb-f472-4c76-aa17-151115629af0","Type":"ContainerStarted","Data":"405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1"}
Oct 30 20:05:41.594723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:41.594175    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7" event={"ID":"9a651deb-f472-4c76-aa17-151115629af0","Type":"ContainerStarted","Data":"b91e88269d4f76ff891624ad92de0b37da21ead78b4f0d037a7ce55e927040e8"}
Oct 30 20:05:41.648940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:41.648869    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7" podStartSLOduration=2.648845744 podStartE2EDuration="2.648845744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:41.646398891 +0000 UTC m=+1866.147102614" watchObservedRunningTime="2026-10-30 20:05:41.648845744 +0000 UTC m=+1866.149549473"
Oct 30 20:05:42.300672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:42.300638    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:42.605320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:42.605227    4519 generic.go:356] "Generic (PLEG): container finished" podID="db80b810-4764-4663-839e-1bb464bafcf8" containerID="321e539d07349d1f39cb41042023fe2899ce5778dae3df5c99e02132eb2c47d2" exitCode=0
Oct 30 20:05:42.605320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:42.605298    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" event={"ID":"db80b810-4764-4663-839e-1bb464bafcf8","Type":"ContainerDied","Data":"321e539d07349d1f39cb41042023fe2899ce5778dae3df5c99e02132eb2c47d2"}
Oct 30 20:05:42.605895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:42.605736    4519 scope.go:122] "RemoveContainer" containerID="321e539d07349d1f39cb41042023fe2899ce5778dae3df5c99e02132eb2c47d2"
Oct 30 20:05:42.607770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:42.607736    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:42.607870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:42.607813    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:42.607870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:42.607856    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:43.037999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:43.037957    4519 scope.go:122] "RemoveContainer" containerID="bb6a02a206b285ceb21f27d8dc75b0751f86c7f841caea2b4c228778ccb725c0"
Oct 30 20:05:43.054727 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:43.054584    4519 scope.go:122] "RemoveContainer" containerID="d1c9cd48fbfbf1c726f3581d9ace569a070fe3be4844d3fb8f223438c42e8f90"
Oct 30 20:05:43.617671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:43.617625    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" event={"ID":"db80b810-4764-4663-839e-1bb464bafcf8","Type":"ContainerStarted","Data":"a237fb21281537ecef3ae72cd09d7f5e1e79c024992fc18884dbe5f12884485d"}
Oct 30 20:05:44.427065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:44.427028    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:05:44.427315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:44.427285    4519 patch_prober.go:29] interesting pod/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 20:05:44.427416 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:44.427376    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 20:05:45.428257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:45.428211    4519 patch_prober.go:29] interesting pod/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 20:05:45.428783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:45.428284    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 20:05:46.427759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:46.427712    4519 patch_prober.go:29] interesting pod/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 20:05:46.427990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:46.427798    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 20:05:47.428109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:47.428066    4519 patch_prober.go:29] interesting pod/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 20:05:47.428608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:47.428124    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 20:05:48.429966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:48.429924    4519 patch_prober.go:29] interesting pod/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 20:05:48.431590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:48.431535    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 20:05:49.430071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:49.430026    4519 patch_prober.go:29] interesting pod/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 20:05:49.431694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:49.431649    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 20:05:49.933775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:49.933739    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:05:50.303196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:50.303072    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:50.358200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:50.358157    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:05:50.433447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:50.433411    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:05:50.888855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:50.888813    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:05:51.093217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:05:51.093185    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:06:02.235971 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:06:02.235922    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:06:10.917231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.917185    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"]
Oct 30 20:06:10.918338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.918305    4519 cpu_manager.go:414] "containerMap: removing container" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy"
Oct 30 20:06:10.918338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.918339    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247c9ec7-c64a-4e57-96a1-0a26a7cf1a37" containerName="istio-proxy"
Oct 30 20:06:10.923336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.923290    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:10.946904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.946862    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"]
Oct 30 20:06:10.950819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.950777    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58d72c\"/\"openshift-service-ca.crt\""
Oct 30 20:06:10.951178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.951154    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58d72c\"/\"kube-root-ca.crt\""
Oct 30 20:06:10.951459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.951444    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58d72c\"/\"test-model-58d72c-kserve-self-signed-certs\""
Oct 30 20:06:10.952904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:10.952881    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58d72c\"/\"default-dockercfg-qmrnb\""
Oct 30 20:06:11.013650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.013602    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667ed976-af33-4e4b-b644-b7358acb616e-tls-certs\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.013873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.013669    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-home\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.013873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.013734    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-dshm\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.013873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.013774    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-model-cache\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.014116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.014086    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-tmp-dir\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.014268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.014249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jkr\" (UniqueName: \"kubernetes.io/projected/667ed976-af33-4e4b-b644-b7358acb616e-kube-api-access-s4jkr\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.116678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.116630    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-tmp-dir\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.116919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.116775    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jkr\" (UniqueName: \"kubernetes.io/projected/667ed976-af33-4e4b-b644-b7358acb616e-kube-api-access-s4jkr\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.117102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.117052    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667ed976-af33-4e4b-b644-b7358acb616e-tls-certs\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.117249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.117128    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-home\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.117249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.117158    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-dshm\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.117249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.117199    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-model-cache\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.117426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.117401    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-tmp-dir\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.117710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.117690    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-model-cache\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.117812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.117714    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-home\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.127919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.127876    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-dshm\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.128175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.128154    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667ed976-af33-4e4b-b644-b7358acb616e-tls-certs\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.159497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.156421    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jkr\" (UniqueName: \"kubernetes.io/projected/667ed976-af33-4e4b-b644-b7358acb616e-kube-api-access-s4jkr\") pod \"test-model-58d72c-kserve-b489cdc57-mzh98\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") " pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.260407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.251273    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:11.890520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:11.886283    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"]
Oct 30 20:06:12.131427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:12.131318    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98" event={"ID":"667ed976-af33-4e4b-b644-b7358acb616e","Type":"ContainerStarted","Data":"07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339"}
Oct 30 20:06:12.131427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:12.131380    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98" event={"ID":"667ed976-af33-4e4b-b644-b7358acb616e","Type":"ContainerStarted","Data":"93db12053bffbd6465b8252532e5549df3f3b5cdefeb3df0d552fb91edb51776"}
Oct 30 20:06:12.173926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:12.173850    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98" podStartSLOduration=2.17382688 podStartE2EDuration="2.17382688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.172418535 +0000 UTC m=+1896.673122259" watchObservedRunningTime="2026-10-30 20:06:12.17382688 +0000 UTC m=+1896.674530601"
Oct 30 20:06:12.252425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:12.252379    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:20.989943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:20.989902    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"]
Oct 30 20:06:20.994891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:20.994850    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.014251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.014220    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"]
Oct 30 20:06:21.018998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.018965    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6cd2ad\"/\"kube-root-ca.crt\""
Oct 30 20:06:21.019211 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.019085    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6cd2ad\"/\"openshift-service-ca.crt\""
Oct 30 20:06:21.025006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.024952    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6cd2ad\"/\"test-model-6cd2ad-kserve-self-signed-certs\""
Oct 30 20:06:21.025006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.024978    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6cd2ad\"/\"default-dockercfg-7xlcp\""
Oct 30 20:06:21.107543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.107446    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-model-cache\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.107781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.107757    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-home\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.108005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.107986    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-dshm\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.108080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.108019    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-tmp-dir\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.108080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.108061    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337d35e-be84-44cd-8b2e-556ff645482e-tls-certs\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.108194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.108122    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7lkb\" (UniqueName: \"kubernetes.io/projected/6337d35e-be84-44cd-8b2e-556ff645482e-kube-api-access-j7lkb\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.209992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.209951    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-home\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.210202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210185    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-dshm\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.210269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210216    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-tmp-dir\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.210269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210250    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337d35e-be84-44cd-8b2e-556ff645482e-tls-certs\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.210383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210288    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7lkb\" (UniqueName: \"kubernetes.io/projected/6337d35e-be84-44cd-8b2e-556ff645482e-kube-api-access-j7lkb\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.210499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210387    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-model-cache\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.210566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210458    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-home\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.212731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210679    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-tmp-dir\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.212731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.210814    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-model-cache\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.212995 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.212875    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-dshm\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.213871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.213843    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337d35e-be84-44cd-8b2e-556ff645482e-tls-certs\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.256545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.256330    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:21.279497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.277273    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7lkb\" (UniqueName: \"kubernetes.io/projected/6337d35e-be84-44cd-8b2e-556ff645482e-kube-api-access-j7lkb\") pod \"test-model-6cd2ad-kserve-7fbcdd6875-sfhf9\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") " pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.327496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.318968    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:21.552584 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.552273    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:21.786713 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:06:21.786673    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6337d35e_be84_44cd_8b2e_556ff645482e.slice/crio-b636a2253188cfe98e62e798860969ef001600bb153545b986003b4cbb2748cf WatchSource:0}: Error finding container b636a2253188cfe98e62e798860969ef001600bb153545b986003b4cbb2748cf: Status 404 returned error can't find the container with id b636a2253188cfe98e62e798860969ef001600bb153545b986003b4cbb2748cf
Oct 30 20:06:21.787133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:21.787105    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"]
Oct 30 20:06:22.399509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:22.399447    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9" event={"ID":"6337d35e-be84-44cd-8b2e-556ff645482e","Type":"ContainerStarted","Data":"e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611"}
Oct 30 20:06:22.418608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:22.418554    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9" event={"ID":"6337d35e-be84-44cd-8b2e-556ff645482e","Type":"ContainerStarted","Data":"b636a2253188cfe98e62e798860969ef001600bb153545b986003b4cbb2748cf"}
Oct 30 20:06:22.507744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:22.507374    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:06:22.568653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:22.568574    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9" podStartSLOduration=2.56854989 podStartE2EDuration="2.56854989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:22.454773575 +0000 UTC m=+1906.955477323" watchObservedRunningTime="2026-10-30 20:06:22.56854989 +0000 UTC m=+1907.069253614"
Oct 30 20:06:23.327660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:23.327600    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:29.744150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.744105    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"]
Oct 30 20:06:29.764491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.763964    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.786004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.785962    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:06:29.786238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.786120    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:29.786963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.786934    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"]
Oct 30 20:06:29.810250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.810046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.810250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.810249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.810586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.810319    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghk7j\" (UniqueName: \"kubernetes.io/projected/66fafaa7-b169-4bee-935e-fcf1d219fefc-kube-api-access-ghk7j\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.914024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.913968    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.915310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.915276    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.921510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.920453    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.921932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.921903    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghk7j\" (UniqueName: \"kubernetes.io/projected/66fafaa7-b169-4bee-935e-fcf1d219fefc-kube-api-access-ghk7j\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.933664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.924281    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:29.981019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:29.980980    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghk7j\" (UniqueName: \"kubernetes.io/projected/66fafaa7-b169-4bee-935e-fcf1d219fefc-kube-api-access-ghk7j\") pod \"maas-api-revoke-sub-79c74a4b-d47wr\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:30.113877 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:30.113784    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:30.528361 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:06:30.528318    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fafaa7_b169_4bee_935e_fcf1d219fefc.slice/crio-53642f243448b86c7f4321781adf94512b6291c21c5582b2cd2d06b4285b9e99 WatchSource:0}: Error finding container 53642f243448b86c7f4321781adf94512b6291c21c5582b2cd2d06b4285b9e99: Status 404 returned error can't find the container with id 53642f243448b86c7f4321781adf94512b6291c21c5582b2cd2d06b4285b9e99
Oct 30 20:06:30.531316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:30.531279    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"]
Oct 30 20:06:30.943771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:30.943726    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"]
Oct 30 20:06:30.994602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:30.994566    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:30.995017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:30.994984    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"]
Oct 30 20:06:31.037018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.036979    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ebe57b\"/\"test-model-ebe57b-kserve-self-signed-certs\""
Oct 30 20:06:31.037349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.037333    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ebe57b\"/\"default-dockercfg-k68vx\""
Oct 30 20:06:31.037586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.037569    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ebe57b\"/\"kube-root-ca.crt\""
Oct 30 20:06:31.037799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.037785    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ebe57b\"/\"openshift-service-ca.crt\""
Oct 30 20:06:31.108996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.108948    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-home\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.109232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.109090    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-dshm\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.109232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.109126    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjhg\" (UniqueName: \"kubernetes.io/projected/a0f21867-bf7c-4226-b673-bae8ad29600b-kube-api-access-gqjhg\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.109357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.109299    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-tmp-dir\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.109357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.109341    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-model-cache\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.109492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.109424    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f21867-bf7c-4226-b673-bae8ad29600b-tls-certs\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.212516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.211454    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-home\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.212516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.211540    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-dshm\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.212516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.211570    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjhg\" (UniqueName: \"kubernetes.io/projected/a0f21867-bf7c-4226-b673-bae8ad29600b-kube-api-access-gqjhg\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.212516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.211651    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-tmp-dir\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.212516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.211682    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-model-cache\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.212516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.211732    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f21867-bf7c-4226-b673-bae8ad29600b-tls-certs\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.215538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.215142    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-home\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.215538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.215378    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-model-cache\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.215538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.215446    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-tmp-dir\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.220709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.220635    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f21867-bf7c-4226-b673-bae8ad29600b-tls-certs\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.221744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.221595    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-dshm\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.263923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.263711    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjhg\" (UniqueName: \"kubernetes.io/projected/a0f21867-bf7c-4226-b673-bae8ad29600b-kube-api-access-gqjhg\") pod \"test-model-ebe57b-kserve-67d88b46dc-zvllh\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") " pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.322511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.322451    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:31.333686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.333251    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:31.348601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.345144    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:31.395652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.395534    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"]
Oct 30 20:06:31.415394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.410108    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.435501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.434899    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d3c508-openshift-default-dockercfg-qmnlc\""
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539018    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-envoy\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539128    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istiod-ca-cert\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539344    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-podinfo\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539582    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-credential-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539728    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-token\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539774    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539873    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjdd\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-kube-api-access-dxjdd\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539930    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-certs\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.630509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.539972    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-data\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.685561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.645852    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-credential-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.744571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.667980    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-credential-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.744964 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.744883    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-token\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.745557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.745512    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.775354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.754576    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-socket\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.815735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.789684    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjdd\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-kube-api-access-dxjdd\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.815735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.789788    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-certs\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.815735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.789833    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-data\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.815735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.789936    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-envoy\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.815735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.790034    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istiod-ca-cert\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.815735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.790300    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-podinfo\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.830490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.817843    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istiod-ca-cert\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.991556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.822113    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-certs\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:31.991556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:31.822692    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-data\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:32.274688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.273637    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"]
Oct 30 20:06:32.355100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.354963    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr" event={"ID":"66fafaa7-b169-4bee-935e-fcf1d219fefc","Type":"ContainerStarted","Data":"e9f3a4142fade18c30f276f3f601bb7de3f195bf008721957c207dc467a9e62c"}
Oct 30 20:06:32.355100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.355032    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr" event={"ID":"66fafaa7-b169-4bee-935e-fcf1d219fefc","Type":"ContainerStarted","Data":"53642f243448b86c7f4321781adf94512b6291c21c5582b2cd2d06b4285b9e99"}
Oct 30 20:06:32.375562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.371637    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:06:32.404951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.404878    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-envoy\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:32.405781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.405728    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-token\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:32.408404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.408363    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-podinfo\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:32.409751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.409695    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjdd\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-kube-api-access-dxjdd\") pod \"e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") " pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:32.414796 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:06:32.414753    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:06:32.611645 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:06:32.611590    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0f21867_bf7c_4226_b673_bae8ad29600b.slice/crio-46f4e7f4ca77eb4dbb2d35752fc4d715bcfa50a83d2bb47ec0392c2a14893771 WatchSource:0}: Error finding container 46f4e7f4ca77eb4dbb2d35752fc4d715bcfa50a83d2bb47ec0392c2a14893771: Status 404 returned error can't find the container with id 46f4e7f4ca77eb4dbb2d35752fc4d715bcfa50a83d2bb47ec0392c2a14893771
Oct 30 20:06:32.621032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.620972    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"]
Oct 30 20:06:32.702176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:32.702039    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:33.156166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.153102    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"]
Oct 30 20:06:33.156166 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:06:33.154773    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf004e6f6_8245_4a0e_987f_5ef0b5b05604.slice/crio-d2a9aae6cd2a05433c08c9dea001d8d95307099b4cd367051d02a26cd8ebf5f8 WatchSource:0}: Error finding container d2a9aae6cd2a05433c08c9dea001d8d95307099b4cd367051d02a26cd8ebf5f8: Status 404 returned error can't find the container with id d2a9aae6cd2a05433c08c9dea001d8d95307099b4cd367051d02a26cd8ebf5f8
Oct 30 20:06:33.159827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.159771    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:33.160114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.160072    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:33.160276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.160156    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:33.369761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.369407    4519 generic.go:356] "Generic (PLEG): container finished" podID="66fafaa7-b169-4bee-935e-fcf1d219fefc" containerID="e9f3a4142fade18c30f276f3f601bb7de3f195bf008721957c207dc467a9e62c" exitCode=0
Oct 30 20:06:33.369761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.369559    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr" event={"ID":"66fafaa7-b169-4bee-935e-fcf1d219fefc","Type":"ContainerDied","Data":"e9f3a4142fade18c30f276f3f601bb7de3f195bf008721957c207dc467a9e62c"}
Oct 30 20:06:33.375520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.374212    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" event={"ID":"f004e6f6-8245-4a0e-987f-5ef0b5b05604","Type":"ContainerStarted","Data":"02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9"}
Oct 30 20:06:33.375520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.374307    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" event={"ID":"f004e6f6-8245-4a0e-987f-5ef0b5b05604","Type":"ContainerStarted","Data":"d2a9aae6cd2a05433c08c9dea001d8d95307099b4cd367051d02a26cd8ebf5f8"}
Oct 30 20:06:33.380751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.380696    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh" event={"ID":"a0f21867-bf7c-4226-b673-bae8ad29600b","Type":"ContainerStarted","Data":"fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201"}
Oct 30 20:06:33.381049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.381021    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh" event={"ID":"a0f21867-bf7c-4226-b673-bae8ad29600b","Type":"ContainerStarted","Data":"46f4e7f4ca77eb4dbb2d35752fc4d715bcfa50a83d2bb47ec0392c2a14893771"}
Oct 30 20:06:33.435081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.434532    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podStartSLOduration=2.4344502439999998 podStartE2EDuration="2.434450244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:33.425944808 +0000 UTC m=+1917.926648532" watchObservedRunningTime="2026-10-30 20:06:33.434450244 +0000 UTC m=+1917.935153969"
Oct 30 20:06:33.899593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.899561    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:33.972668 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:33.972633    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:34.021594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.021510    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh" podStartSLOduration=4.021480071 podStartE2EDuration="4.021480071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:33.488818904 +0000 UTC m=+1917.989522631" watchObservedRunningTime="2026-10-30 20:06:34.021480071 +0000 UTC m=+1918.522183788"
Oct 30 20:06:34.076104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.076054    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fafaa7-b169-4bee-935e-fcf1d219fefc-kube-api-access-ghk7j\" (UniqueName: \"kubernetes.io/projected/66fafaa7-b169-4bee-935e-fcf1d219fefc-kube-api-access-ghk7j\") pod \"66fafaa7-b169-4bee-935e-fcf1d219fefc\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") "
Oct 30 20:06:34.076424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.076129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-xks-ca\") pod \"66fafaa7-b169-4bee-935e-fcf1d219fefc\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") "
Oct 30 20:06:34.076580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.076529    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-openshift-ca\") pod \"66fafaa7-b169-4bee-935e-fcf1d219fefc\" (UID: \"66fafaa7-b169-4bee-935e-fcf1d219fefc\") "
Oct 30 20:06:34.078399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.078361    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-openshift-ca" pod "66fafaa7-b169-4bee-935e-fcf1d219fefc" (UID: "66fafaa7-b169-4bee-935e-fcf1d219fefc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:34.083785 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.083744    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fafaa7-b169-4bee-935e-fcf1d219fefc-kube-api-access-ghk7j" pod "66fafaa7-b169-4bee-935e-fcf1d219fefc" (UID: "66fafaa7-b169-4bee-935e-fcf1d219fefc"). InnerVolumeSpecName "kube-api-access-ghk7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:34.092530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.092459    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-xks-ca" pod "66fafaa7-b169-4bee-935e-fcf1d219fefc" (UID: "66fafaa7-b169-4bee-935e-fcf1d219fefc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:34.178978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.178936    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:06:34.178978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.178979    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghk7j\" (UniqueName: \"kubernetes.io/projected/66fafaa7-b169-4bee-935e-fcf1d219fefc-kube-api-access-ghk7j\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:06:34.179544 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.178993    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fafaa7-b169-4bee-935e-fcf1d219fefc-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:06:34.334380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.334344    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:34.394967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.394933    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"
Oct 30 20:06:34.394967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.394956    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr" event={"ID":"66fafaa7-b169-4bee-935e-fcf1d219fefc","Type":"ContainerDied","Data":"53642f243448b86c7f4321781adf94512b6291c21c5582b2cd2d06b4285b9e99"}
Oct 30 20:06:34.395241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.395015    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53642f243448b86c7f4321781adf94512b6291c21c5582b2cd2d06b4285b9e99"
Oct 30 20:06:34.782584 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:34.781808    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:35.409271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:35.409234    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:35.411187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:35.411152    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:06:39.459227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:39.459180    4519 generic.go:356] "Generic (PLEG): container finished" podID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerID="da9cf93be5dba0da7c8924357effb264738b1470d96d8cc910d42f9bfa31f2e1" exitCode=0
Oct 30 20:06:39.459881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:39.459221    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" event={"ID":"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e","Type":"ContainerDied","Data":"da9cf93be5dba0da7c8924357effb264738b1470d96d8cc910d42f9bfa31f2e1"}
Oct 30 20:06:39.459960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:39.459893    4519 scope.go:122] "RemoveContainer" containerID="da9cf93be5dba0da7c8924357effb264738b1470d96d8cc910d42f9bfa31f2e1"
Oct 30 20:06:39.462227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:39.462188    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:39.462384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:39.462294    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:39.462384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:39.462329    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:40.485961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.485912    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" event={"ID":"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e","Type":"ContainerStarted","Data":"0970ee28848e15352cf9aed9479408d376a8fe78b50c5b4a4e84baa90aafd290"}
Oct 30 20:06:40.606106 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.606065    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:06:40.607226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.607195    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 20:06:40.607383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.607250    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 20:06:40.878552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.870358    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"]
Oct 30 20:06:40.878552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.871563    4519 cpu_manager.go:414] "containerMap: removing container" podUID="66fafaa7-b169-4bee-935e-fcf1d219fefc" containerName="revoke-keys"
Oct 30 20:06:40.878552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.871591    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fafaa7-b169-4bee-935e-fcf1d219fefc" containerName="revoke-keys"
Oct 30 20:06:40.878552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.871795    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fafaa7-b169-4bee-935e-fcf1d219fefc" containerName="revoke-keys"
Oct 30 20:06:40.914492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.914243    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:40.935172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.932748    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"]
Oct 30 20:06:40.940123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:40.940093    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a64be2-openshift-default-dockercfg-zbjbc\""
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089060    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-credential-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089171    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-envoy\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089202    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-certs\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089327    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726bac57-6740-4469-bb0e-5a073dac1490-istiod-ca-cert\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089383    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-data\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089418    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726bac57-6740-4469-bb0e-5a073dac1490-istio-podinfo\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089528    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-istio-token\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089599    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.091498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.089632    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7chv\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-kube-api-access-w7chv\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.199740    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.199797    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7chv\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-kube-api-access-w7chv\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.199972    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-credential-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.200057    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-envoy\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.200085    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-certs\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.200226    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726bac57-6740-4469-bb0e-5a073dac1490-istiod-ca-cert\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.200283    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-data\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.200319    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726bac57-6740-4469-bb0e-5a073dac1490-istio-podinfo\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.200506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.200411    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-istio-token\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.352886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.281449    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.449139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.290145    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-certs\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.449139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.290719    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-credential-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.449139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.291319    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-data\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.449139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.292032    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726bac57-6740-4469-bb0e-5a073dac1490-istiod-ca-cert\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.500764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.461990    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:41.515229 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.514355    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726bac57-6740-4469-bb0e-5a073dac1490-istio-podinfo\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.553510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.545129    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:41.553892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.553835    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7chv\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-kube-api-access-w7chv\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.618497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.612099    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-istio-token\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.641558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.620367    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 20:06:41.641558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.633397    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 20:06:41.641558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.621982    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-envoy\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:41.697493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.696511    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:06:41.790943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:41.776160    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:42.317049 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:06:42.317007    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726bac57_6740_4469_bb0e_5a073dac1490.slice/crio-b872d057014f52ed3da85c25371c04b07172b7881416986c5327eeefe50a111b WatchSource:0}: Error finding container b872d057014f52ed3da85c25371c04b07172b7881416986c5327eeefe50a111b: Status 404 returned error can't find the container with id b872d057014f52ed3da85c25371c04b07172b7881416986c5327eeefe50a111b
Oct 30 20:06:42.317254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.317148    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"]
Oct 30 20:06:42.322147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.322098    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:42.322379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.322357    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:42.322549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.322530    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:42.612551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.612068    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 20:06:42.612551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.612156    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 20:06:42.628542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.625658    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" event={"ID":"726bac57-6740-4469-bb0e-5a073dac1490","Type":"ContainerStarted","Data":"4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e"}
Oct 30 20:06:42.628542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.625730    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" event={"ID":"726bac57-6740-4469-bb0e-5a073dac1490","Type":"ContainerStarted","Data":"b872d057014f52ed3da85c25371c04b07172b7881416986c5327eeefe50a111b"}
Oct 30 20:06:42.797111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:42.797026    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:43.131491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.124638    4519 scope.go:122] "RemoveContainer" containerID="b2396c7df785453f210a988a2abe6c802f3f65bb206e759edd647675e2cb7f21"
Oct 30 20:06:43.215033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.214976    4519 scope.go:122] "RemoveContainer" containerID="e7d6c908d04381609a257d198240d34eac55ae164f8a24515a75f281954242dd"
Oct 30 20:06:43.228629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.228601    4519 scope.go:122] "RemoveContainer" containerID="b8846c446639c744c3c5f5b19e9abac64acdd66556549e033919ce72b7cbf49f"
Oct 30 20:06:43.259717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.259621    4519 scope.go:122] "RemoveContainer" containerID="f5ec55756893f30542d8ec4773ece8f3c1873944615dba3ffe127c8fa79178d3"
Oct 30 20:06:43.606407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.606347    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 20:06:43.606702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.606418    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 20:06:43.799670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.799621    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:43.861478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:43.861324    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podStartSLOduration=3.861301545 podStartE2EDuration="3.861301545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:42.682872062 +0000 UTC m=+1927.183575948" watchObservedRunningTime="2026-10-30 20:06:43.861301545 +0000 UTC m=+1928.362005270"
Oct 30 20:06:44.606684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:44.606639    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 20:06:44.606911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:44.606719    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 20:06:44.652679 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:44.652633    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:44.656885 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:44.656845    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:06:45.606326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:45.606274    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 20:06:45.606902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:45.606344    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 20:06:46.606668 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:46.606623    4519 patch_prober.go:29] interesting pod/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 20:06:46.607175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:46.606709    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 20:06:47.481499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:47.480749    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:06:47.760438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:47.760335    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:06:47.762300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:47.762261    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:06:49.994419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:49.994313    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"]
Oct 30 20:06:50.000889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.000862    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.022842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.022800    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"]
Oct 30 20:06:50.026752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.026674    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:06:50.027174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.027148    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:50.112214 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.112174    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.112427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.112291    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rvz9\" (UniqueName: \"kubernetes.io/projected/47dc1ed1-71ff-46e4-882c-a6776b0eae15-kube-api-access-6rvz9\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.112510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.112498    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.214034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.213993    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.214218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.214083    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rvz9\" (UniqueName: \"kubernetes.io/projected/47dc1ed1-71ff-46e4-882c-a6776b0eae15-kube-api-access-6rvz9\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.214261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.214246    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.214955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.214927    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.216611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.216591    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.234362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.234329    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rvz9\" (UniqueName: \"kubernetes.io/projected/47dc1ed1-71ff-46e4-882c-a6776b0eae15-kube-api-access-6rvz9\") pod \"maas-api-revoke-sub-245312d4-q6ck5\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.319250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.319159    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:50.673715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.667611    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"]
Oct 30 20:06:50.790639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:50.790594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5" event={"ID":"47dc1ed1-71ff-46e4-882c-a6776b0eae15","Type":"ContainerStarted","Data":"8aefe4712e3c9770577a72df44e298041b526517427c47b219093e47ce6a9126"}
Oct 30 20:06:51.803645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:51.803601    4519 generic.go:356] "Generic (PLEG): container finished" podID="47dc1ed1-71ff-46e4-882c-a6776b0eae15" containerID="b2363595975e9f1dacba23a7475ac677558173ef52e01e32a17d772c8f49e2da" exitCode=0
Oct 30 20:06:51.804193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:51.803690    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5" event={"ID":"47dc1ed1-71ff-46e4-882c-a6776b0eae15","Type":"ContainerDied","Data":"b2363595975e9f1dacba23a7475ac677558173ef52e01e32a17d772c8f49e2da"}
Oct 30 20:06:53.022560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.022522    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:53.186949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.186902    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-xks-ca\") pod \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") "
Oct 30 20:06:53.187185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.186975    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dc1ed1-71ff-46e4-882c-a6776b0eae15-kube-api-access-6rvz9\" (UniqueName: \"kubernetes.io/projected/47dc1ed1-71ff-46e4-882c-a6776b0eae15-kube-api-access-6rvz9\") pod \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") "
Oct 30 20:06:53.187185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.187123    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-openshift-ca\") pod \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\" (UID: \"47dc1ed1-71ff-46e4-882c-a6776b0eae15\") "
Oct 30 20:06:53.187587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.187550    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-openshift-ca" pod "47dc1ed1-71ff-46e4-882c-a6776b0eae15" (UID: "47dc1ed1-71ff-46e4-882c-a6776b0eae15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:53.188200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.188171    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:06:53.189415 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.189381    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-xks-ca" pod "47dc1ed1-71ff-46e4-882c-a6776b0eae15" (UID: "47dc1ed1-71ff-46e4-882c-a6776b0eae15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:53.189743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.189721    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dc1ed1-71ff-46e4-882c-a6776b0eae15-kube-api-access-6rvz9" pod "47dc1ed1-71ff-46e4-882c-a6776b0eae15" (UID: "47dc1ed1-71ff-46e4-882c-a6776b0eae15"). InnerVolumeSpecName "kube-api-access-6rvz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:53.290547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.290499    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dc1ed1-71ff-46e4-882c-a6776b0eae15-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:06:53.290547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.290541    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rvz9\" (UniqueName: \"kubernetes.io/projected/47dc1ed1-71ff-46e4-882c-a6776b0eae15-kube-api-access-6rvz9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:06:53.825117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.825070    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"
Oct 30 20:06:53.825371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.825056    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5" event={"ID":"47dc1ed1-71ff-46e4-882c-a6776b0eae15","Type":"ContainerDied","Data":"8aefe4712e3c9770577a72df44e298041b526517427c47b219093e47ce6a9126"}
Oct 30 20:06:53.825371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:06:53.825233    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aefe4712e3c9770577a72df44e298041b526517427c47b219093e47ce6a9126"
Oct 30 20:07:02.229221 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:07:02.229178    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:07:02.303979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.303930    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"]
Oct 30 20:07:02.304930 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.304909    4519 cpu_manager.go:414] "containerMap: removing container" podUID="47dc1ed1-71ff-46e4-882c-a6776b0eae15" containerName="revoke-keys"
Oct 30 20:07:02.304998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.304931    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47dc1ed1-71ff-46e4-882c-a6776b0eae15" containerName="revoke-keys"
Oct 30 20:07:02.305095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.305085    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="47dc1ed1-71ff-46e4-882c-a6776b0eae15" containerName="revoke-keys"
Oct 30 20:07:02.308925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.308900    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.329218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.329178    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"]
Oct 30 20:07:02.334794 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.334499    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:02.334794 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.334612    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:07:02.457264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.457218    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6qg\" (UniqueName: \"kubernetes.io/projected/283434f1-16fa-42d1-a988-2443df18643a-kube-api-access-kr6qg\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.457498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.457430    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283434f1-16fa-42d1-a988-2443df18643a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.457564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.457530    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283434f1-16fa-42d1-a988-2443df18643a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.561863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.561746    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6qg\" (UniqueName: \"kubernetes.io/projected/283434f1-16fa-42d1-a988-2443df18643a-kube-api-access-kr6qg\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.562076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.561957    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283434f1-16fa-42d1-a988-2443df18643a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.562076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.562044    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283434f1-16fa-42d1-a988-2443df18643a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.563292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.563255    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283434f1-16fa-42d1-a988-2443df18643a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.565150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.565122    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283434f1-16fa-42d1-a988-2443df18643a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.586069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.586028    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6qg\" (UniqueName: \"kubernetes.io/projected/283434f1-16fa-42d1-a988-2443df18643a-kube-api-access-kr6qg\") pod \"maas-api-revoke-sub-9d1d6b1a-5xr2w\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.626014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.625972    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:02.949341 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:07:02.948691    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283434f1_16fa_42d1_a988_2443df18643a.slice/crio-4951a8da4732be08ab7f4f0142f61ad86df64d9f46f2500da5934cad954f818f WatchSource:0}: Error finding container 4951a8da4732be08ab7f4f0142f61ad86df64d9f46f2500da5934cad954f818f: Status 404 returned error can't find the container with id 4951a8da4732be08ab7f4f0142f61ad86df64d9f46f2500da5934cad954f818f
Oct 30 20:07:02.952125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:02.952047    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"]
Oct 30 20:07:03.052550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:03.052501    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w" event={"ID":"283434f1-16fa-42d1-a988-2443df18643a","Type":"ContainerStarted","Data":"4951a8da4732be08ab7f4f0142f61ad86df64d9f46f2500da5934cad954f818f"}
Oct 30 20:07:04.133492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:04.123336    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w" event={"ID":"283434f1-16fa-42d1-a988-2443df18643a","Type":"ContainerStarted","Data":"c1f8bfab21580d670e8907e6683277f73c92b231b757543f9e2fc2a06527137a"}
Oct 30 20:07:04.196493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:04.196009    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w" podStartSLOduration=2.195986209 podStartE2EDuration="2.195986209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:04.195276413 +0000 UTC m=+1948.695980137" watchObservedRunningTime="2026-10-30 20:07:04.195986209 +0000 UTC m=+1948.696689935"
Oct 30 20:07:05.210494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:05.204895    4519 generic.go:356] "Generic (PLEG): container finished" podID="283434f1-16fa-42d1-a988-2443df18643a" containerID="c1f8bfab21580d670e8907e6683277f73c92b231b757543f9e2fc2a06527137a" exitCode=0
Oct 30 20:07:05.210494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:05.205171    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w" event={"ID":"283434f1-16fa-42d1-a988-2443df18643a","Type":"ContainerDied","Data":"c1f8bfab21580d670e8907e6683277f73c92b231b757543f9e2fc2a06527137a"}
Oct 30 20:07:06.520048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.520011    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:06.615046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.615005    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283434f1-16fa-42d1-a988-2443df18643a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283434f1-16fa-42d1-a988-2443df18643a-maas-api-openshift-ca\") pod \"283434f1-16fa-42d1-a988-2443df18643a\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") "
Oct 30 20:07:06.615276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.615117    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283434f1-16fa-42d1-a988-2443df18643a-kube-api-access-kr6qg\" (UniqueName: \"kubernetes.io/projected/283434f1-16fa-42d1-a988-2443df18643a-kube-api-access-kr6qg\") pod \"283434f1-16fa-42d1-a988-2443df18643a\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") "
Oct 30 20:07:06.615413 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.615394    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283434f1-16fa-42d1-a988-2443df18643a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283434f1-16fa-42d1-a988-2443df18643a-maas-api-xks-ca\") pod \"283434f1-16fa-42d1-a988-2443df18643a\" (UID: \"283434f1-16fa-42d1-a988-2443df18643a\") "
Oct 30 20:07:06.615526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.615500    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283434f1-16fa-42d1-a988-2443df18643a-maas-api-openshift-ca" pod "283434f1-16fa-42d1-a988-2443df18643a" (UID: "283434f1-16fa-42d1-a988-2443df18643a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:06.616386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.616353    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283434f1-16fa-42d1-a988-2443df18643a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:06.617925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.617893    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283434f1-16fa-42d1-a988-2443df18643a-maas-api-xks-ca" pod "283434f1-16fa-42d1-a988-2443df18643a" (UID: "283434f1-16fa-42d1-a988-2443df18643a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:06.618048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.617947    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283434f1-16fa-42d1-a988-2443df18643a-kube-api-access-kr6qg" pod "283434f1-16fa-42d1-a988-2443df18643a" (UID: "283434f1-16fa-42d1-a988-2443df18643a"). InnerVolumeSpecName "kube-api-access-kr6qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:06.718800 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.718753    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283434f1-16fa-42d1-a988-2443df18643a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:06.719053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:06.718836    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6qg\" (UniqueName: \"kubernetes.io/projected/283434f1-16fa-42d1-a988-2443df18643a-kube-api-access-kr6qg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:07.232685 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:07.232636    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w" event={"ID":"283434f1-16fa-42d1-a988-2443df18643a","Type":"ContainerDied","Data":"4951a8da4732be08ab7f4f0142f61ad86df64d9f46f2500da5934cad954f818f"}
Oct 30 20:07:07.232895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:07.232707    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4951a8da4732be08ab7f4f0142f61ad86df64d9f46f2500da5934cad954f818f"
Oct 30 20:07:07.232895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:07.232783    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"
Oct 30 20:07:15.028262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.028216    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"]
Oct 30 20:07:15.029363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.029332    4519 cpu_manager.go:414] "containerMap: removing container" podUID="283434f1-16fa-42d1-a988-2443df18643a" containerName="revoke-keys"
Oct 30 20:07:15.029363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.029364    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283434f1-16fa-42d1-a988-2443df18643a" containerName="revoke-keys"
Oct 30 20:07:15.029627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.029598    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="283434f1-16fa-42d1-a988-2443df18643a" containerName="revoke-keys"
Oct 30 20:07:15.033915 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.033881    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.056667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.056150    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"]
Oct 30 20:07:15.063155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.058567    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:07:15.063155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.058671    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:15.175517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.174429    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.175517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.174772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878h2\" (UniqueName: \"kubernetes.io/projected/a1afbaac-ae49-46fe-be86-ca2098a32aed-kube-api-access-878h2\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.175517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.175115    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.277406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.277346    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.277675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.277428    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.277772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.277719    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878h2\" (UniqueName: \"kubernetes.io/projected/a1afbaac-ae49-46fe-be86-ca2098a32aed-kube-api-access-878h2\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.278963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.278875    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.285409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.285367    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.360493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.356494    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-878h2\" (UniqueName: \"kubernetes.io/projected/a1afbaac-ae49-46fe-be86-ca2098a32aed-kube-api-access-878h2\") pod \"maas-api-revoke-keys-e2e-inf-a-0506c9-22l27\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.377031 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.376977    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:15.784761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:15.784721    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"]
Oct 30 20:07:15.787680 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:07:15.787642    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1afbaac_ae49_46fe_be86_ca2098a32aed.slice/crio-6c91bc5d698c03e275191e063e65c366c00f09f1d90937d1e921c842a0b0f5bb WatchSource:0}: Error finding container 6c91bc5d698c03e275191e063e65c366c00f09f1d90937d1e921c842a0b0f5bb: Status 404 returned error can't find the container with id 6c91bc5d698c03e275191e063e65c366c00f09f1d90937d1e921c842a0b0f5bb
Oct 30 20:07:16.427445 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.427397    4519 generic.go:356] "Generic (PLEG): container finished" podID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerID="8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa" exitCode=0
Oct 30 20:07:16.428105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.427514    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" event={"ID":"6bb1f85f-6380-4563-88e9-b55d8cb73967","Type":"ContainerDied","Data":"8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa"}
Oct 30 20:07:16.428193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.428175    4519 scope.go:122] "RemoveContainer" containerID="8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa"
Oct 30 20:07:16.430394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.430365    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27" event={"ID":"a1afbaac-ae49-46fe-be86-ca2098a32aed","Type":"ContainerStarted","Data":"ab38e5a4a1f59a07d5edfc00108284c9cd43ccded23da8d9c3f586f751f2e41d"}
Oct 30 20:07:16.430547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.430435    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27" event={"ID":"a1afbaac-ae49-46fe-be86-ca2098a32aed","Type":"ContainerStarted","Data":"6c91bc5d698c03e275191e063e65c366c00f09f1d90937d1e921c842a0b0f5bb"}
Oct 30 20:07:16.430871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.430839    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:16.430960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.430915    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:16.430960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.430949    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:16.538304 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:16.538234    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27" podStartSLOduration=2.538213266 podStartE2EDuration="2.538213266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:16.533792726 +0000 UTC m=+1961.034496451" watchObservedRunningTime="2026-10-30 20:07:16.538213266 +0000 UTC m=+1961.038916989"
Oct 30 20:07:17.442128 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:17.442081    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" event={"ID":"6bb1f85f-6380-4563-88e9-b55d8cb73967","Type":"ContainerStarted","Data":"77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5"}
Oct 30 20:07:17.444221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:17.444188    4519 generic.go:356] "Generic (PLEG): container finished" podID="a1afbaac-ae49-46fe-be86-ca2098a32aed" containerID="ab38e5a4a1f59a07d5edfc00108284c9cd43ccded23da8d9c3f586f751f2e41d" exitCode=0
Oct 30 20:07:17.444381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:17.444268    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27" event={"ID":"a1afbaac-ae49-46fe-be86-ca2098a32aed","Type":"ContainerDied","Data":"ab38e5a4a1f59a07d5edfc00108284c9cd43ccded23da8d9c3f586f751f2e41d"}
Oct 30 20:07:17.710510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:17.710403    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:07:17.711732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:17.711700    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 20:07:17.711858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:17.711762    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 20:07:18.710764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.710323    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 20:07:18.710764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.710396    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 20:07:18.808518 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.808392    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:18.913968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.911880    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1afbaac-ae49-46fe-be86-ca2098a32aed-kube-api-access-878h2\" (UniqueName: \"kubernetes.io/projected/a1afbaac-ae49-46fe-be86-ca2098a32aed-kube-api-access-878h2\") pod \"a1afbaac-ae49-46fe-be86-ca2098a32aed\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") "
Oct 30 20:07:18.913968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.912060    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-openshift-ca\") pod \"a1afbaac-ae49-46fe-be86-ca2098a32aed\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") "
Oct 30 20:07:18.913968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.912363    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-xks-ca\") pod \"a1afbaac-ae49-46fe-be86-ca2098a32aed\" (UID: \"a1afbaac-ae49-46fe-be86-ca2098a32aed\") "
Oct 30 20:07:18.913968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.912741    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-openshift-ca" pod "a1afbaac-ae49-46fe-be86-ca2098a32aed" (UID: "a1afbaac-ae49-46fe-be86-ca2098a32aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:18.913968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.913514    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:18.916399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.914802    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1afbaac-ae49-46fe-be86-ca2098a32aed-kube-api-access-878h2" pod "a1afbaac-ae49-46fe-be86-ca2098a32aed" (UID: "a1afbaac-ae49-46fe-be86-ca2098a32aed"). InnerVolumeSpecName "kube-api-access-878h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:18.916399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:18.915639    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-xks-ca" pod "a1afbaac-ae49-46fe-be86-ca2098a32aed" (UID: "a1afbaac-ae49-46fe-be86-ca2098a32aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:19.015277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.015226    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1afbaac-ae49-46fe-be86-ca2098a32aed-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:19.015277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.015273    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878h2\" (UniqueName: \"kubernetes.io/projected/a1afbaac-ae49-46fe-be86-ca2098a32aed-kube-api-access-878h2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:19.495932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.495871    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27" event={"ID":"a1afbaac-ae49-46fe-be86-ca2098a32aed","Type":"ContainerDied","Data":"6c91bc5d698c03e275191e063e65c366c00f09f1d90937d1e921c842a0b0f5bb"}
Oct 30 20:07:19.495932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.495907    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"
Oct 30 20:07:19.496286 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.495982    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c91bc5d698c03e275191e063e65c366c00f09f1d90937d1e921c842a0b0f5bb"
Oct 30 20:07:19.668495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.666615    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:07:19.710620 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.710577    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 20:07:19.710823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:19.710653    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 20:07:20.710548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:20.710446    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 20:07:20.710774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:20.710565    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 20:07:21.710824 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:21.710777    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 20:07:21.711308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:21.710851    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 20:07:22.710594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:22.710559    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 20:07:22.710892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:22.710629    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 20:07:23.716144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:23.716101    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:07:23.717397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:23.717368    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:07:32.276007 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:07:32.275964    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:07:43.371826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:43.371768    4519 scope.go:122] "RemoveContainer" containerID="5e765b318da18d9ab8564cc12414a6364ee6c083efa2187cd7a36186d8faaaa0"
Oct 30 20:07:43.387587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:43.387549    4519 scope.go:122] "RemoveContainer" containerID="1aad72e0de40626fa19233d3d3cea026c9469a5088ecbd69f5bfac4975501dc3"
Oct 30 20:07:43.400390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:43.400360    4519 scope.go:122] "RemoveContainer" containerID="c3c874d1a808264bf7342faad54cbefa52ae1fd8a7dea6d342b890cdd68d6ccd"
Oct 30 20:07:46.164631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:46.164546    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"]
Oct 30 20:07:46.165227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:46.164893    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy" containerID="cri-o://ae6132e449f1c4200ebd6a277e76cef1f7fb1416baeae845d48efc8adf78dbfc" gracePeriod=30
Oct 30 20:07:51.915165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:51.915116    4519 generic.go:356] "Generic (PLEG): container finished" podID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerID="ae6132e449f1c4200ebd6a277e76cef1f7fb1416baeae845d48efc8adf78dbfc" exitCode=0
Oct 30 20:07:51.915611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:51.915182    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" event={"ID":"79d5626f-14c3-45e7-8e08-e732f45b5316","Type":"ContainerDied","Data":"ae6132e449f1c4200ebd6a277e76cef1f7fb1416baeae845d48efc8adf78dbfc"}
Oct 30 20:07:51.915611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:51.915241    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr" event={"ID":"79d5626f-14c3-45e7-8e08-e732f45b5316","Type":"ContainerDied","Data":"372565fc46ff2a87fc444d2de20d2e46aafecf018094d902334eb604819b623e"}
Oct 30 20:07:51.915611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:51.915259    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="372565fc46ff2a87fc444d2de20d2e46aafecf018094d902334eb604819b623e"
Oct 30 20:07:51.932995 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:51.932956    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 20:07:52.066000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.065892    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-socket\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066037    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d5626f-14c3-45e7-8e08-e732f45b5316-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d5626f-14c3-45e7-8e08-e732f45b5316-istiod-ca-cert\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-envoy\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066191    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-kube-api-access-swfr5\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-kube-api-access-swfr5\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066270    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-socket" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:52.066353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066338    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-token\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-token\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066372    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-certs\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066440    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-data\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066599    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-podinfo\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.066667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066646    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d5626f-14c3-45e7-8e08-e732f45b5316-istiod-ca-cert" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:52.066704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.066686    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-credential-socket\") pod \"79d5626f-14c3-45e7-8e08-e732f45b5316\" (UID: \"79d5626f-14c3-45e7-8e08-e732f45b5316\") "
Oct 30 20:07:52.067311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.067275    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-data" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:52.067552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.067526    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-certs" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:52.067653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.067593    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-credential-socket" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:52.068683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.068522    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.068683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.068612    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.068683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.068627    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d5626f-14c3-45e7-8e08-e732f45b5316-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.068683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.068641    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.068683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.068655    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.069609 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.069576    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-envoy" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:52.070065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.070034    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-token" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:52.072000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.071965    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-kube-api-access-swfr5" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "kube-api-access-swfr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:52.072118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.072007    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-podinfo" pod "79d5626f-14c3-45e7-8e08-e732f45b5316" (UID: "79d5626f-14c3-45e7-8e08-e732f45b5316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:07:52.170940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.170892    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.170940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.170932    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfr5\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-kube-api-access-swfr5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.170940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.170944    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.170940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.170955    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d5626f-14c3-45e7-8e08-e732f45b5316-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:07:52.923936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.923893    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"
Oct 30 20:07:52.992107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.992017    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"]
Oct 30 20:07:52.997873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:52.997834    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-q22dr"]
Oct 30 20:07:53.616230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:53.616192    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" path="/var/lib/kubelet/pods/79d5626f-14c3-45e7-8e08-e732f45b5316/volumes"
Oct 30 20:07:59.070123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:59.070078    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"]
Oct 30 20:07:59.077367 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:59.077315    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0506c9-22l27"]
Oct 30 20:07:59.615036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:07:59.614990    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1afbaac-ae49-46fe-be86-ca2098a32aed" path="/var/lib/kubelet/pods/a1afbaac-ae49-46fe-be86-ca2098a32aed/volumes"
Oct 30 20:08:01.427774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.427719    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"]
Oct 30 20:08:01.428955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.428923    4519 cpu_manager.go:414] "containerMap: removing container" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy"
Oct 30 20:08:01.428955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.428952    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy"
Oct 30 20:08:01.429191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.429007    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a1afbaac-ae49-46fe-be86-ca2098a32aed" containerName="revoke-keys"
Oct 30 20:08:01.429191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.429013    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1afbaac-ae49-46fe-be86-ca2098a32aed" containerName="revoke-keys"
Oct 30 20:08:01.429191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.429148    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d5626f-14c3-45e7-8e08-e732f45b5316" containerName="istio-proxy"
Oct 30 20:08:01.429191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.429168    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1afbaac-ae49-46fe-be86-ca2098a32aed" containerName="revoke-keys"
Oct 30 20:08:01.433119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.433089    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.475134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.475077    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"]
Oct 30 20:08:01.479981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.479920    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"]
Oct 30 20:08:01.485907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.485867    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.500571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.500530    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"]
Oct 30 20:08:01.541207 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541161    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.541207 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541215    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.541541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541326    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.541541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541379    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.541632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541604    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.541672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541648    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.541742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541724    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.541841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541827    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.541950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541935    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.542014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.541998    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.542134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542110    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.542287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542153    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfxg\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-kube-api-access-6sfxg\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.542340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542292    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.542542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542499    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.542752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542543    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b060d-29e9-4e2d-bf09-6f9d9b177082-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.542752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542641    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.542752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542684    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-kube-api-access-bsvj8\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.542891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.542785    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.644418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644360    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.644721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644529    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.644721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644560    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.644721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644585    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.644721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644617    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.644721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644695    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.644721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644718    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644747    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644837    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644904    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.644924    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645013    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645022    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645048    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645066    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645118    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645154    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfxg\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-kube-api-access-6sfxg\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645203    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645247    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645334    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645355    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645387    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b060d-29e9-4e2d-bf09-6f9d9b177082-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645394    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.645565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645454    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.646078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645578    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-kube-api-access-bsvj8\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.646078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.645355    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.646200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.646155    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.646278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.646254    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b060d-29e9-4e2d-bf09-6f9d9b177082-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.647887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.647855    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.648043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.647961    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.648043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.647980    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.648625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.648584    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.700684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.700592    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.704736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.704694    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.704736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.704722    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfxg\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-kube-api-access-6sfxg\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:01.704941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.704698    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-kube-api-access-bsvj8\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.755037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.754970    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:01.810229 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:01.810177    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:02.104736 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:08:02.104699    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157b060d_29e9_4e2d_bf09_6f9d9b177082.slice/crio-27afc64d66b00546191f058c15b87947bf17dbb2aae3c25423752fad297e33bc WatchSource:0}: Error finding container 27afc64d66b00546191f058c15b87947bf17dbb2aae3c25423752fad297e33bc: Status 404 returned error can't find the container with id 27afc64d66b00546191f058c15b87947bf17dbb2aae3c25423752fad297e33bc
Oct 30 20:08:02.106670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.106637    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"]
Oct 30 20:08:02.108884 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.108842    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:02.108984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.108928    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:02.108984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.108959    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:02.152181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.152136    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" event={"ID":"157b060d-29e9-4e2d-bf09-6f9d9b177082","Type":"ContainerStarted","Data":"27afc64d66b00546191f058c15b87947bf17dbb2aae3c25423752fad297e33bc"}
Oct 30 20:08:02.160825 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:08:02.160767    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5039f7_483c_4f5f_b5fb_79cacbe254fb.slice/crio-0afeb32d8553e98aee0c67665e91ff94280f42eeda72122f02a5e1e7378de182 WatchSource:0}: Error finding container 0afeb32d8553e98aee0c67665e91ff94280f42eeda72122f02a5e1e7378de182: Status 404 returned error can't find the container with id 0afeb32d8553e98aee0c67665e91ff94280f42eeda72122f02a5e1e7378de182
Oct 30 20:08:02.161017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.160869    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"]
Oct 30 20:08:02.176265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.176206    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:02.176447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.176320    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:02.176447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:02.176366    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:02.235402 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:08:02.235361    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:08:03.197397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.197346    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" event={"ID":"157b060d-29e9-4e2d-bf09-6f9d9b177082","Type":"ContainerStarted","Data":"0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a"}
Oct 30 20:08:03.200119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.200066    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" event={"ID":"ba5039f7-483c-4f5f-b5fb-79cacbe254fb","Type":"ContainerStarted","Data":"ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf"}
Oct 30 20:08:03.200366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.200167    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" event={"ID":"ba5039f7-483c-4f5f-b5fb-79cacbe254fb","Type":"ContainerStarted","Data":"0afeb32d8553e98aee0c67665e91ff94280f42eeda72122f02a5e1e7378de182"}
Oct 30 20:08:03.279494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.264373    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podStartSLOduration=2.264344943 podStartE2EDuration="2.264344943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:03.256072127 +0000 UTC m=+2007.756775881" watchObservedRunningTime="2026-10-30 20:08:03.264344943 +0000 UTC m=+2007.765048670"
Oct 30 20:08:03.327703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.327065    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podStartSLOduration=2.327036839 podStartE2EDuration="2.327036839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:03.325522107 +0000 UTC m=+2007.826225829" watchObservedRunningTime="2026-10-30 20:08:03.327036839 +0000 UTC m=+2007.827740564"
Oct 30 20:08:03.799121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.799084    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:03.800159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.799501    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 20:08:03.800435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.800402    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 20:08:03.830216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.830119    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:03.832029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.831990    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 20:08:03.832248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:03.832228    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 20:08:04.758039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:04.757992    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 20:08:04.759403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:04.758904    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 20:08:04.813497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:04.812990    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 20:08:04.813497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:04.813073    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 20:08:05.765497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:05.763001    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 20:08:05.765497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:05.763083    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 20:08:05.811494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:05.811386    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 20:08:05.811750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:05.811545    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 20:08:06.765495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:06.758024    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 20:08:06.765495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:06.758097    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 20:08:06.811495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:06.811192    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 20:08:06.811495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:06.811282    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 20:08:07.197497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.196969    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"]
Oct 30 20:08:07.197752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.197649    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" containerID="cri-o://dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0" gracePeriod=30
Oct 30 20:08:07.770634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.770582    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 20:08:07.785506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.771308    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 20:08:07.864162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.864098    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:07.881507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.867056    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 20:08:07.881507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.867124    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 20:08:07.944637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.944585    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"]
Oct 30 20:08:07.983947 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.983909    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:07.999369 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:07.999315    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"]
Oct 30 20:08:08.029675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.029371    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:08.042088 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.042054    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:08:08.114046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.108931    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8sw\" (UniqueName: \"kubernetes.io/projected/3f6a21c6-bc67-438c-916d-c46e58357286-kube-api-access-wl8sw\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.114046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.109165    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.114046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.109356    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.221675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.219960    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.221675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.220244    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8sw\" (UniqueName: \"kubernetes.io/projected/3f6a21c6-bc67-438c-916d-c46e58357286-kube-api-access-wl8sw\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.221675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.220410    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.221675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.220898    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.227425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.223303    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.241346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.241308    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8sw\" (UniqueName: \"kubernetes.io/projected/3f6a21c6-bc67-438c-916d-c46e58357286-kube-api-access-wl8sw\") pod \"maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.417542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.415694    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:08.771082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.770964    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 20:08:08.823714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.771819    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 20:08:08.824504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.818582    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 20:08:08.824788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:08.824750    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 20:08:09.116184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.116130    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"]
Oct 30 20:08:09.758406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.758223    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 20:08:09.758406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.758308    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 20:08:09.811437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.811318    4519 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 20:08:09.811437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.811398    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 20:08:09.861552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.860786    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7" event={"ID":"3f6a21c6-bc67-438c-916d-c46e58357286","Type":"ContainerStarted","Data":"272a3560738d300825719b656ddf5421accf3c4ff63fd74c1f061e633f705728"}
Oct 30 20:08:09.861552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.860870    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7" event={"ID":"3f6a21c6-bc67-438c-916d-c46e58357286","Type":"ContainerStarted","Data":"c90b7f026b9ea0577e3409c8bd00d2a766ec180710237b37b13bd56f28d7a155"}
Oct 30 20:08:09.913789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:09.913701    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7" podStartSLOduration=2.913677826 podStartE2EDuration="2.913677826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:09.908406801 +0000 UTC m=+2014.409110525" watchObservedRunningTime="2026-10-30 20:08:09.913677826 +0000 UTC m=+2014.414381551"
Oct 30 20:08:10.764883 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.764766    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:10.765375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.765338    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:10.766411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.766386    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:08:10.817347 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.817314    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:10.817884 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.817742    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:10.818674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.818649    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:08:10.875072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.875023    4519 generic.go:356] "Generic (PLEG): container finished" podID="3f6a21c6-bc67-438c-916d-c46e58357286" containerID="272a3560738d300825719b656ddf5421accf3c4ff63fd74c1f061e633f705728" exitCode=0
Oct 30 20:08:10.875316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:10.875069    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7" event={"ID":"3f6a21c6-bc67-438c-916d-c46e58357286","Type":"ContainerDied","Data":"272a3560738d300825719b656ddf5421accf3c4ff63fd74c1f061e633f705728"}
Oct 30 20:08:12.083023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.082983    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:12.107382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.107339    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-xks-ca\") pod \"3f6a21c6-bc67-438c-916d-c46e58357286\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") "
Oct 30 20:08:12.107629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.107453    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-openshift-ca\") pod \"3f6a21c6-bc67-438c-916d-c46e58357286\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") "
Oct 30 20:08:12.107769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.107752    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6a21c6-bc67-438c-916d-c46e58357286-kube-api-access-wl8sw\" (UniqueName: \"kubernetes.io/projected/3f6a21c6-bc67-438c-916d-c46e58357286-kube-api-access-wl8sw\") pod \"3f6a21c6-bc67-438c-916d-c46e58357286\" (UID: \"3f6a21c6-bc67-438c-916d-c46e58357286\") "
Oct 30 20:08:12.107878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.107853    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-openshift-ca" pod "3f6a21c6-bc67-438c-916d-c46e58357286" (UID: "3f6a21c6-bc67-438c-916d-c46e58357286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:12.108830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.108784    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:12.110178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.110137    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6a21c6-bc67-438c-916d-c46e58357286-kube-api-access-wl8sw" pod "3f6a21c6-bc67-438c-916d-c46e58357286" (UID: "3f6a21c6-bc67-438c-916d-c46e58357286"). InnerVolumeSpecName "kube-api-access-wl8sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:12.110178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.110163    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-xks-ca" pod "3f6a21c6-bc67-438c-916d-c46e58357286" (UID: "3f6a21c6-bc67-438c-916d-c46e58357286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:12.211202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.211147    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6a21c6-bc67-438c-916d-c46e58357286-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:12.211202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.211191    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8sw\" (UniqueName: \"kubernetes.io/projected/3f6a21c6-bc67-438c-916d-c46e58357286-kube-api-access-wl8sw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:12.264495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.259707    4519 patch_prober.go:29] interesting pod/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:08:12.264495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.259789    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:08:12.850219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.850178    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 20:08:12.902342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.902293    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"
Oct 30 20:08:12.902342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.902309    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7" event={"ID":"3f6a21c6-bc67-438c-916d-c46e58357286","Type":"ContainerDied","Data":"c90b7f026b9ea0577e3409c8bd00d2a766ec180710237b37b13bd56f28d7a155"}
Oct 30 20:08:12.902659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.902379    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c90b7f026b9ea0577e3409c8bd00d2a766ec180710237b37b13bd56f28d7a155"
Oct 30 20:08:12.905014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.904980    4519 generic.go:356] "Generic (PLEG): container finished" podID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerID="dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0" exitCode=0
Oct 30 20:08:12.905215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.905020    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" event={"ID":"5378b4fd-98d4-4fa8-88c3-31b19c705b45","Type":"ContainerDied","Data":"dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0"}
Oct 30 20:08:12.905215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.905051    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"
Oct 30 20:08:12.905215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.905098    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw" event={"ID":"5378b4fd-98d4-4fa8-88c3-31b19c705b45","Type":"ContainerDied","Data":"012afa46b5eac4b5f9d6ba5bef4c1d9028660551aa2bc2145f36fee064911806"}
Oct 30 20:08:12.905215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.905128    4519 scope.go:122] "RemoveContainer" containerID="dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0"
Oct 30 20:08:12.919805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.919773    4519 scope.go:122] "RemoveContainer" containerID="b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d"
Oct 30 20:08:12.935019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.934977    4519 scope.go:122] "RemoveContainer" containerID="dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0"
Oct 30 20:08:12.935719 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:08:12.935688    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0\": container with ID starting with dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0 not found: ID does not exist" containerID="dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0"
Oct 30 20:08:12.935841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.935727    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0"} err="failed to get container status \"dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0\": rpc error: code = NotFound desc = could not find container \"dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0\": container with ID starting with dceda31d8edfc572c1c4dfba509af12da0df21f7f32a5e1d0c678747dbafc9b0 not found: ID does not exist"
Oct 30 20:08:12.935841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.935750    4519 scope.go:122] "RemoveContainer" containerID="b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d"
Oct 30 20:08:12.936134 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:08:12.936109    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d\": container with ID starting with b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d not found: ID does not exist" containerID="b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d"
Oct 30 20:08:12.936205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:12.936138    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d"} err="failed to get container status \"b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d\": rpc error: code = NotFound desc = could not find container \"b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d\": container with ID starting with b48d14acd0f293395939b098d294c05800fb1d2d60756032d1a0f6c974719e1d not found: ID does not exist"
Oct 30 20:08:13.035438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.035391    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-envoy\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.035699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.035543    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-socket\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.035699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.035601    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istiod-ca-cert\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.035835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.035701    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-data\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.035835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.035765    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-certs\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.035954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.035841    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-socket" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:13.035954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.035927    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-token\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-token\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.036070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.036045    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-credential-socket\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.036131 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.036059    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-data" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:13.036131 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.036078    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-certs" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:13.036222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.036134    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istiod-ca-cert" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:13.036222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.036171    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-podinfo\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.036299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.036281    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-credential-socket" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:13.036342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.036306    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-kube-api-access-hb74r\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-kube-api-access-hb74r\") pod \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\" (UID: \"5378b4fd-98d4-4fa8-88c3-31b19c705b45\") "
Oct 30 20:08:13.037262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.037231    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.037262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.037263    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.037510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.037279    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.037510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.037293    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.037510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.037306    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.038281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.038255    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-podinfo" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:08:13.038616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.038447    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-envoy" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:13.038674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.038499    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-kube-api-access-hb74r" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "kube-api-access-hb74r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:13.038674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.038507    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-token" pod "5378b4fd-98d4-4fa8-88c3-31b19c705b45" (UID: "5378b4fd-98d4-4fa8-88c3-31b19c705b45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:13.139691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.139572    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.139691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.139611    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.139691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.139621    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb74r\" (UniqueName: \"kubernetes.io/projected/5378b4fd-98d4-4fa8-88c3-31b19c705b45-kube-api-access-hb74r\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.139691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.139632    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5378b4fd-98d4-4fa8-88c3-31b19c705b45-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:08:13.268745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.268699    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"]
Oct 30 20:08:13.274681 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.274501    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0506c9-openshift-default-57ff8d466f-tvqrw"]
Oct 30 20:08:13.615479 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:13.615433    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" path="/var/lib/kubelet/pods/5378b4fd-98d4-4fa8-88c3-31b19c705b45/volumes"
Oct 30 20:08:16.102690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:16.102631    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:16.104497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:16.104021    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:16.110507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:16.108827    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:08:16.129491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:16.127193    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:16.153154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:16.150276    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:16.153154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:16.151267    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:16.392788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:16.392634    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh" podUID="9de8bd51-bab6-409e-aa03-9a40668303a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.692512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:30.692414    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:08:30.697129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:30.693052    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.587728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:31.587667    4519 generic.go:356] "Generic (PLEG): container finished" podID="726bac57-6740-4469-bb0e-5a073dac1490" containerID="4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e" exitCode=0
Oct 30 20:08:31.587949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:31.587741    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" event={"ID":"726bac57-6740-4469-bb0e-5a073dac1490","Type":"ContainerDied","Data":"4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e"}
Oct 30 20:08:31.588450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:31.588429    4519 scope.go:122] "RemoveContainer" containerID="4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e"
Oct 30 20:08:31.591069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:31.591033    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:31.591218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:31.591133    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:31.591218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:31.591191    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:32.232255 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:08:32.232213    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:08:32.601212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:32.601123    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" event={"ID":"726bac57-6740-4469-bb0e-5a073dac1490","Type":"ContainerStarted","Data":"50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842"}
Oct 30 20:08:32.791750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:32.791699    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:08:32.792005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:32.791846    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 20:08:32.792005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:32.791894    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 20:08:33.791975 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:33.791925    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 20:08:33.792575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:33.792005    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 20:08:34.797511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:34.791930    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 20:08:34.797511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:34.792013    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 20:08:35.411293 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.411245    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:35.411724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.411694    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:35.702561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.702454    4519 generic.go:356] "Generic (PLEG): container finished" podID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerID="02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9" exitCode=0
Oct 30 20:08:35.703157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.703124    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" event={"ID":"f004e6f6-8245-4a0e-987f-5ef0b5b05604","Type":"ContainerDied","Data":"02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9"}
Oct 30 20:08:35.703984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.703957    4519 scope.go:122] "RemoveContainer" containerID="02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9"
Oct 30 20:08:35.709377 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.709334    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:35.709541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.709433    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:35.709541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.709518    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:35.796783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.796722    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 20:08:35.797010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:35.796833    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 20:08:36.715854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:36.715808    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" event={"ID":"f004e6f6-8245-4a0e-987f-5ef0b5b05604","Type":"ContainerStarted","Data":"46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4"}
Oct 30 20:08:36.792238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:36.792184    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 20:08:36.792482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:36.792269    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 20:08:37.705494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:37.705057    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:08:37.705494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:37.705226    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:37.705494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:37.705290    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:37.798197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:37.798143    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 20:08:37.799005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:37.798949    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 20:08:38.712998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:38.712949    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:38.713378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:38.713336    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:38.796038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:38.795982    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 20:08:38.796284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:38.796065    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 20:08:39.702516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:39.702442    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:39.708979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:39.708925    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:40.233271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:40.233216    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:08:40.234132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:40.234096    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:08:40.237171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:40.237139    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:08:40.703036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:40.703000    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:40.703577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:40.703547    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:41.703544 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:41.703449    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:41.704032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:41.703571    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:42.702947 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:42.702895    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:42.703164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:42.702994    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:43.492124 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:43.492080    4519 scope.go:122] "RemoveContainer" containerID="c9885dfc37087e1afa46c4e107ef484a070309d081a26d472d7ceefaaae9b283"
Oct 30 20:08:43.528575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:43.528528    4519 scope.go:122] "RemoveContainer" containerID="ae6132e449f1c4200ebd6a277e76cef1f7fb1416baeae845d48efc8adf78dbfc"
Oct 30 20:08:43.703440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:43.703392    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 20:08:43.703661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:43.703480    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 20:08:44.709024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:44.708984    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:08:44.709532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:44.709503    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:08:44.710511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:08:44.710456    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:09:00.532251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:00.532198    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"]
Oct 30 20:09:00.532936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:00.532506    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy" containerID="cri-o://d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815" gracePeriod=30
Oct 30 20:09:01.005934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:01.005874    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"]
Oct 30 20:09:01.006200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:01.006116    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy" containerID="cri-o://8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c" gracePeriod=30
Oct 30 20:09:02.231679 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:09:02.231615    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:09:05.927941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:05.927896    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 20:09:06.072136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072088    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istiod-ca-cert\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.072389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072159    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-data\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.072389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072213    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-kube-api-access-qfbqs\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-kube-api-access-qfbqs\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.072389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072340    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-certs\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.072389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072385    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-socket\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.072742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072669    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-envoy\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.072742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072688    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-data" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.072742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072722    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-podinfo\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.073036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.072998    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-credential-socket\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.073151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.073129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-token\") pod \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\" (UID: \"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4\") "
Oct 30 20:09:06.075122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.074096    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.075122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.073005    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-certs" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.075122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.073923    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istiod-ca-cert" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:06.075122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.074703    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-socket" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.075122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.075084    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-credential-socket" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.081657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.081586    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-envoy" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.082074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.082038    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-token" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:06.084127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.084072    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-podinfo" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:09:06.095807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.095730    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-kube-api-access-qfbqs" pod "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" (UID: "a6f478e2-4b31-4dba-99ff-b96cecc9b1d4"). InnerVolumeSpecName "kube-api-access-qfbqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:06.176669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176613    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.176669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176670    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.176960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176685    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.176960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176702    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.176960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176719    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.176960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176732    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbqs\" (UniqueName: \"kubernetes.io/projected/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-kube-api-access-qfbqs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.176960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176744    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.176960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.176758    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.273145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.273111    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 20:09:06.320634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.320575    4519 generic.go:356] "Generic (PLEG): container finished" podID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerID="d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815" exitCode=0
Oct 30 20:09:06.320887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.320652    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" event={"ID":"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4","Type":"ContainerDied","Data":"d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815"}
Oct 30 20:09:06.320887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.320730    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b" event={"ID":"a6f478e2-4b31-4dba-99ff-b96cecc9b1d4","Type":"ContainerDied","Data":"97f0237f647aa485ea2e1e2a8e1f945f67d2cebb8ec344d2c3d7545141272554"}
Oct 30 20:09:06.320887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.320767    4519 scope.go:122] "RemoveContainer" containerID="d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815"
Oct 30 20:09:06.321010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.320928    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"
Oct 30 20:09:06.323960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.323760    4519 generic.go:356] "Generic (PLEG): container finished" podID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerID="8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c" exitCode=0
Oct 30 20:09:06.323960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.323887    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" event={"ID":"1633beb3-9fe0-49bb-89f1-f5cb464e4dec","Type":"ContainerDied","Data":"8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c"}
Oct 30 20:09:06.323960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.323927    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6" event={"ID":"1633beb3-9fe0-49bb-89f1-f5cb464e4dec","Type":"ContainerDied","Data":"b693225f2c4e02a4d91b24c24fa5243f0705a4478f60525e96f52b9aa1a22019"}
Oct 30 20:09:06.324274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.323992    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"
Oct 30 20:09:06.347300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.347243    4519 scope.go:122] "RemoveContainer" containerID="d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815"
Oct 30 20:09:06.350331 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:09:06.350288    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815\": container with ID starting with d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815 not found: ID does not exist" containerID="d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815"
Oct 30 20:09:06.350542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.350342    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815"} err="failed to get container status \"d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815\": rpc error: code = NotFound desc = could not find container \"d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815\": container with ID starting with d3145fbf2f097480bfc3a1411c3f12cdf0fe6c70496b141196ba3bcd3a736815 not found: ID does not exist"
Oct 30 20:09:06.350542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.350372    4519 scope.go:122] "RemoveContainer" containerID="8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c"
Oct 30 20:09:06.367102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.367061    4519 scope.go:122] "RemoveContainer" containerID="8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c"
Oct 30 20:09:06.367758 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:09:06.367718    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c\": container with ID starting with 8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c not found: ID does not exist" containerID="8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c"
Oct 30 20:09:06.367927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.367766    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c"} err="failed to get container status \"8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c\": rpc error: code = NotFound desc = could not find container \"8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c\": container with ID starting with 8c909d9cdfbb3f1f7702e225b7456150b3ac3064746db246f23546db925c304c not found: ID does not exist"
Oct 30 20:09:06.381567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.381512    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-podinfo\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.381976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.381953    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-certs\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.382142 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382124    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-envoy\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.382401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382360    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-certs" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.382750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382498    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-token\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-token\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.382750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382562    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-kube-api-access-958xx\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-kube-api-access-958xx\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.382750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382607    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-data\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.383013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382766    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-credential-socket\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.383013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382842    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-socket\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.383013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.382968    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istiod-ca-cert\") pod \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\" (UID: \"1633beb3-9fe0-49bb-89f1-f5cb464e4dec\") "
Oct 30 20:09:06.384010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.383980    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.384453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.384415    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-credential-socket" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.384826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.384799    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-socket" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.384935 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.384740    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istiod-ca-cert" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:06.385045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.385021    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-data" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.387878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.387829    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-token" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:06.387878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.387859    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-kube-api-access-958xx" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "kube-api-access-958xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:06.388088 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.387887    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-envoy" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:06.389563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.389531    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-podinfo" pod "1633beb3-9fe0-49bb-89f1-f5cb464e4dec" (UID: "1633beb3-9fe0-49bb-89f1-f5cb464e4dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:09:06.416638 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.416594    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"]
Oct 30 20:09:06.424879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.424773    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-rhq8b"]
Oct 30 20:09:06.486398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486344    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.486398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486394    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.486696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486410    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.486696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486424    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.486696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486438    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.486696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486449    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.486696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486479    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-958xx\" (UniqueName: \"kubernetes.io/projected/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-kube-api-access-958xx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.486696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.486494    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1633beb3-9fe0-49bb-89f1-f5cb464e4dec-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:06.694191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.694046    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"]
Oct 30 20:09:06.700261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:06.700210    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bd6n6"]
Oct 30 20:09:07.663279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:07.663230    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" path="/var/lib/kubelet/pods/1633beb3-9fe0-49bb-89f1-f5cb464e4dec/volumes"
Oct 30 20:09:07.664364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:07.664333    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" path="/var/lib/kubelet/pods/a6f478e2-4b31-4dba-99ff-b96cecc9b1d4/volumes"
Oct 30 20:09:10.399766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:10.399714    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"]
Oct 30 20:09:10.424933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:10.424890    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb16ce-72mg7"]
Oct 30 20:09:11.615284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:11.615241    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6a21c6-bc67-438c-916d-c46e58357286" path="/var/lib/kubelet/pods/3f6a21c6-bc67-438c-916d-c46e58357286/volumes"
Oct 30 20:09:18.742615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:18.742568    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"]
Oct 30 20:09:18.743166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:18.742891    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy" containerID="cri-o://e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba" gracePeriod=30
Oct 30 20:09:20.529294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.529248    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"]
Oct 30 20:09:20.530109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530088    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy"
Oct 30 20:09:20.530109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530110    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy"
Oct 30 20:09:20.530252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530130    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy"
Oct 30 20:09:20.530252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530136    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy"
Oct 30 20:09:20.530252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530149    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3f6a21c6-bc67-438c-916d-c46e58357286" containerName="revoke-keys"
Oct 30 20:09:20.530252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530154    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6a21c6-bc67-438c-916d-c46e58357286" containerName="revoke-keys"
Oct 30 20:09:20.530252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530178    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy"
Oct 30 20:09:20.530252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530182    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy"
Oct 30 20:09:20.530605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530297    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy"
Oct 30 20:09:20.530605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530309    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f478e2-4b31-4dba-99ff-b96cecc9b1d4" containerName="istio-proxy"
Oct 30 20:09:20.530605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530318    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6a21c6-bc67-438c-916d-c46e58357286" containerName="revoke-keys"
Oct 30 20:09:20.530605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.530328    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1633beb3-9fe0-49bb-89f1-f5cb464e4dec" containerName="istio-proxy"
Oct 30 20:09:20.543135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.543086    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.566164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.566104    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"]
Oct 30 20:09:20.568808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.568772    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c8fa69\"/\"auth-test-model-c8fa69-kserve-self-signed-certs\""
Oct 30 20:09:20.568996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.568916    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c8fa69\"/\"default-dockercfg-lrpqz\""
Oct 30 20:09:20.568996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.568984    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c8fa69\"/\"openshift-service-ca.crt\""
Oct 30 20:09:20.573918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.573886    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c8fa69\"/\"kube-root-ca.crt\""
Oct 30 20:09:20.735542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.735496    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-tmp-dir\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.735806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.735672    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-model-cache\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.735910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.735812    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f25887-41c2-498f-87c9-2299fa9008c2-tls-certs\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.735910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.735893    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-dshm\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.736091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.736076    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-home\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.736182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.736166    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtx5\" (UniqueName: \"kubernetes.io/projected/75f25887-41c2-498f-87c9-2299fa9008c2-kube-api-access-cgtx5\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.839564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.839059    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-dshm\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.839564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.839330    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-home\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.839564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.839441    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtx5\" (UniqueName: \"kubernetes.io/projected/75f25887-41c2-498f-87c9-2299fa9008c2-kube-api-access-cgtx5\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.839941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.839596    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-tmp-dir\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.839941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.839696    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-model-cache\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.839941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.839787    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f25887-41c2-498f-87c9-2299fa9008c2-tls-certs\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.856572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.841156    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-tmp-dir\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.856572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.841442    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-model-cache\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.856572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.847712    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-dshm\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.856572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.848024    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-home\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.860791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.860749    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f25887-41c2-498f-87c9-2299fa9008c2-tls-certs\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:20.903333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:20.903250    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtx5\" (UniqueName: \"kubernetes.io/projected/75f25887-41c2-498f-87c9-2299fa9008c2-kube-api-access-cgtx5\") pod \"auth-test-model-c8fa69-kserve-56f96655b9-pr46x\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") " pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:21.174542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:21.174501    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:21.504356 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:09:21.504296    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f25887_41c2_498f_87c9_2299fa9008c2.slice/crio-f968af5d10edcbae387fca3cb4642918828f717e6d76562618798ce63a343763 WatchSource:0}: Error finding container f968af5d10edcbae387fca3cb4642918828f717e6d76562618798ce63a343763: Status 404 returned error can't find the container with id f968af5d10edcbae387fca3cb4642918828f717e6d76562618798ce63a343763
Oct 30 20:09:21.505805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:21.505752    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"]
Oct 30 20:09:21.550319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:21.550273    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x" event={"ID":"75f25887-41c2-498f-87c9-2299fa9008c2","Type":"ContainerStarted","Data":"f968af5d10edcbae387fca3cb4642918828f717e6d76562618798ce63a343763"}
Oct 30 20:09:22.630861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:22.630800    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x" event={"ID":"75f25887-41c2-498f-87c9-2299fa9008c2","Type":"ContainerStarted","Data":"bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622"}
Oct 30 20:09:22.672555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:22.672082    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x" podStartSLOduration=2.6720621209999997 podStartE2EDuration="2.672062121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:22.670792814 +0000 UTC m=+2087.171496542" watchObservedRunningTime="2026-10-30 20:09:22.672062121 +0000 UTC m=+2087.172765845"
Oct 30 20:09:23.175060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:23.175021    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:24.115575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.115536    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 20:09:24.223694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.223632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-socket\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.223961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.223763    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-podinfo\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.223961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.223889    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-data\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.223961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.223937    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-certs\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.224146 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.223995    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-token\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-token\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.224146 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224054    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-credential-socket\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.224146 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224071    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-socket" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:24.224317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224271    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-envoy\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.224317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224273    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-data" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:24.224458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224428    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-certs" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:24.224561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224512    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-credential-socket" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:24.224623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istiod-ca-cert\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.224734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224713    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-kube-api-access-z5xxl\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-kube-api-access-z5xxl\") pod \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\" (UID: \"11cf99a9-cef2-4e0b-8455-47bbd2c33edf\") "
Oct 30 20:09:24.224891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.224868    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istiod-ca-cert" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:24.225950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.225917    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.226089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.225954    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.226089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.225970    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.226089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.225983    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.226089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.225996    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.226773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.226745    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-token" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:24.226986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.226958    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-podinfo" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:09:24.227440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.227413    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-envoy" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:24.232964 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.232833    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-kube-api-access-z5xxl" pod "11cf99a9-cef2-4e0b-8455-47bbd2c33edf" (UID: "11cf99a9-cef2-4e0b-8455-47bbd2c33edf"). InnerVolumeSpecName "kube-api-access-z5xxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:24.328135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.328038    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5xxl\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-kube-api-access-z5xxl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.328135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.328075    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.328135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.328086    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.328135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.328095    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11cf99a9-cef2-4e0b-8455-47bbd2c33edf-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:24.655365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.655247    4519 generic.go:356] "Generic (PLEG): container finished" podID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerID="e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba" exitCode=0
Oct 30 20:09:24.655365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.655306    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"
Oct 30 20:09:24.655365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.655327    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" event={"ID":"11cf99a9-cef2-4e0b-8455-47bbd2c33edf","Type":"ContainerDied","Data":"e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba"}
Oct 30 20:09:24.655733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.655395    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn" event={"ID":"11cf99a9-cef2-4e0b-8455-47bbd2c33edf","Type":"ContainerDied","Data":"4bece96dfcf34d4801012450f7fddd3471796848cd8d88c78a50ae1b50ba117d"}
Oct 30 20:09:24.655733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.655427    4519 scope.go:122] "RemoveContainer" containerID="e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba"
Oct 30 20:09:24.672819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.672792    4519 scope.go:122] "RemoveContainer" containerID="0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405"
Oct 30 20:09:24.687243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.687203    4519 scope.go:122] "RemoveContainer" containerID="e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba"
Oct 30 20:09:24.687751 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:09:24.687725    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba\": container with ID starting with e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba not found: ID does not exist" containerID="e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba"
Oct 30 20:09:24.687859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.687763    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba"} err="failed to get container status \"e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba\": rpc error: code = NotFound desc = could not find container \"e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba\": container with ID starting with e92f53031e46696a4a2e1fc255c53c516949ce0f2e92e5fcaf7761fd4d08d0ba not found: ID does not exist"
Oct 30 20:09:24.687859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.687787    4519 scope.go:122] "RemoveContainer" containerID="0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405"
Oct 30 20:09:24.688158 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:09:24.688140    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405\": container with ID starting with 0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405 not found: ID does not exist" containerID="0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405"
Oct 30 20:09:24.688214 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.688166    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405"} err="failed to get container status \"0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405\": rpc error: code = NotFound desc = could not find container \"0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405\": container with ID starting with 0a570d162c997ecff1b982e418ca56346948ae3df969e7b6eccea2880b294405 not found: ID does not exist"
Oct 30 20:09:24.711306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.711260    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"]
Oct 30 20:09:24.723382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:24.723343    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-eb16ce-openshift-default-645b45477c-bsttn"]
Oct 30 20:09:25.618242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:25.618199    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" path="/var/lib/kubelet/pods/11cf99a9-cef2-4e0b-8455-47bbd2c33edf/volumes"
Oct 30 20:09:30.564137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:30.564075    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"]
Oct 30 20:09:30.564788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:30.564373    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" containerID="cri-o://d55a6de033dde18f00607a4d8e7ba77e9f7cdc2ac8d8e475efb17e9a9c6e0115" gracePeriod=30
Oct 30 20:09:31.157098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:31.157058    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"]
Oct 30 20:09:31.157348 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:31.157301    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" containerID="cri-o://a1b587dea3e2a0c95487ba83306ab7cfd0cfd242db67f34125f74f44159f3610" gracePeriod=30
Oct 30 20:09:31.174981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:31.174921    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:31.196800 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:31.196747    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:31.767568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:31.767519    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:09:32.225752 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:09:32.225708    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:09:35.295039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:35.294992    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:09:35.295610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:35.295084    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:09:35.581556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:35.580948    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:09:35.581556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:35.581019    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:09:35.796744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:35.796308    4519 generic.go:356] "Generic (PLEG): container finished" podID="b791e0b1-579c-4491-8226-418ad0459043" containerID="d55a6de033dde18f00607a4d8e7ba77e9f7cdc2ac8d8e475efb17e9a9c6e0115" exitCode=0
Oct 30 20:09:35.796744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:35.796505    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" event={"ID":"b791e0b1-579c-4491-8226-418ad0459043","Type":"ContainerDied","Data":"d55a6de033dde18f00607a4d8e7ba77e9f7cdc2ac8d8e475efb17e9a9c6e0115"}
Oct 30 20:09:35.994401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:35.993927    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 20:09:36.045508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.041126    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-socket\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.045508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.041307    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-kube-api-access-ljlxl\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-kube-api-access-ljlxl\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.045508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.041449    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b791e0b1-579c-4491-8226-418ad0459043-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b791e0b1-579c-4491-8226-418ad0459043-istiod-ca-cert\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.046127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.046063    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-certs\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.046495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.046422    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-data\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.056411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.054801    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-socket" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:36.056411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.055221    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b791e0b1-579c-4491-8226-418ad0459043-istiod-ca-cert" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:36.056411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.055492    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-certs" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:36.056411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.056319    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-data" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:36.083658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.083579    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-envoy\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.089663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.084117    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-istio-token\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-istio-token\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.099882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.099623    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b791e0b1-579c-4491-8226-418ad0459043-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b791e0b1-579c-4491-8226-418ad0459043-istio-podinfo\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.117689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.100218    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-credential-socket\") pod \"b791e0b1-579c-4491-8226-418ad0459043\" (UID: \"b791e0b1-579c-4491-8226-418ad0459043\") "
Oct 30 20:09:36.129655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.100523    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-credential-socket" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:36.131583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.131481    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.131583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.131515    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b791e0b1-579c-4491-8226-418ad0459043-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.131583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.131530    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.131583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.131544    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.131583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.131554    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.206312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.170406    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-envoy" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:36.206312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.170667    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-kube-api-access-ljlxl" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "kube-api-access-ljlxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:36.206312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.171638    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-istio-token" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:36.256025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.255762    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b791e0b1-579c-4491-8226-418ad0459043-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.256025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.255801    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.256025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.255817    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlxl\" (UniqueName: \"kubernetes.io/projected/b791e0b1-579c-4491-8226-418ad0459043-kube-api-access-ljlxl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.256025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.255974    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b791e0b1-579c-4491-8226-418ad0459043-istio-podinfo" pod "b791e0b1-579c-4491-8226-418ad0459043" (UID: "b791e0b1-579c-4491-8226-418ad0459043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:09:36.445495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.424103    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b791e0b1-579c-4491-8226-418ad0459043-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:36.867501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.867440    4519 generic.go:356] "Generic (PLEG): container finished" podID="176a9878-2bb1-444d-a865-9149b3061fd1" containerID="a1b587dea3e2a0c95487ba83306ab7cfd0cfd242db67f34125f74f44159f3610" exitCode=0
Oct 30 20:09:36.868014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.867983    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" event={"ID":"176a9878-2bb1-444d-a865-9149b3061fd1","Type":"ContainerDied","Data":"a1b587dea3e2a0c95487ba83306ab7cfd0cfd242db67f34125f74f44159f3610"}
Oct 30 20:09:36.891525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.891483    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4" event={"ID":"b791e0b1-579c-4491-8226-418ad0459043","Type":"ContainerDied","Data":"1998c207744d19bdbc666bba06451d106e51a83469c76384f6f3039d7e42a33d"}
Oct 30 20:09:36.893819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.891844    4519 scope.go:122] "RemoveContainer" containerID="d55a6de033dde18f00607a4d8e7ba77e9f7cdc2ac8d8e475efb17e9a9c6e0115"
Oct 30 20:09:36.907238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:36.898072    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"
Oct 30 20:09:37.166423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.153933    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"]
Oct 30 20:09:37.248557 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:09:37.243578    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb791e0b1_579c_4491_8226_418ad0459043.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:09:37.272649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.253309    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-kmgb4"]
Oct 30 20:09:37.463317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.463287    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 20:09:37.628030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.627920    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/176a9878-2bb1-444d-a865-9149b3061fd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176a9878-2bb1-444d-a865-9149b3061fd1-istio-podinfo\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628053    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-credential-socket\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628095    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-certs\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628224    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-socket\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628477 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628330    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-envoy\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628477 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628389    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-istio-token\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-istio-token\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628477 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628423    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-kube-api-access-xp925\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-kube-api-access-xp925\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628630    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-data\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.628746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.628696    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176a9878-2bb1-444d-a865-9149b3061fd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176a9878-2bb1-444d-a865-9149b3061fd1-istiod-ca-cert\") pod \"176a9878-2bb1-444d-a865-9149b3061fd1\" (UID: \"176a9878-2bb1-444d-a865-9149b3061fd1\") "
Oct 30 20:09:37.629119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.629085    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-socket" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:37.629327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.629292    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-data" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:37.629808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.629785    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176a9878-2bb1-444d-a865-9149b3061fd1-istiod-ca-cert" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:37.630085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.630066    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176a9878-2bb1-444d-a865-9149b3061fd1-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.630159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.630094    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.630159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.630105    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.630706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.630682    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-credential-socket" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:37.631062 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.631034    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-certs" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:37.634510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.634438    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-envoy" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:37.636513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.635889    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-kube-api-access-xp925" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "kube-api-access-xp925". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:37.636513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.636041    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-istio-token" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:37.638004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.637982    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/176a9878-2bb1-444d-a865-9149b3061fd1-istio-podinfo" pod "176a9878-2bb1-444d-a865-9149b3061fd1" (UID: "176a9878-2bb1-444d-a865-9149b3061fd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:09:37.642495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.642432    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b791e0b1-579c-4491-8226-418ad0459043" path="/var/lib/kubelet/pods/b791e0b1-579c-4491-8226-418ad0459043/volumes"
Oct 30 20:09:37.732258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.732210    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.732258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.732254    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.732258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.732266    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp925\" (UniqueName: \"kubernetes.io/projected/176a9878-2bb1-444d-a865-9149b3061fd1-kube-api-access-xp925\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.732719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.732278    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176a9878-2bb1-444d-a865-9149b3061fd1-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.732719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.732289    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:37.732719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:37.732299    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176a9878-2bb1-444d-a865-9149b3061fd1-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:09:38.014492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:38.013569    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m" event={"ID":"176a9878-2bb1-444d-a865-9149b3061fd1","Type":"ContainerDied","Data":"f740761865d3cf2d7af6beba3f788ffe7f03351454c2caddb4a62339486422b6"}
Oct 30 20:09:38.014492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:38.013655    4519 scope.go:122] "RemoveContainer" containerID="a1b587dea3e2a0c95487ba83306ab7cfd0cfd242db67f34125f74f44159f3610"
Oct 30 20:09:38.014492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:38.013848    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"
Oct 30 20:09:38.167200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:38.167155    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"]
Oct 30 20:09:38.174933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:38.174886    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-gfq7m"]
Oct 30 20:09:39.664913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:39.664863    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" path="/var/lib/kubelet/pods/176a9878-2bb1-444d-a865-9149b3061fd1/volumes"
Oct 30 20:09:43.635131 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:09:43.635094    4519 scope.go:122] "RemoveContainer" containerID="ce12c59207d0e435a722225469195a77aa7e2bc93eaf91f8ca5fbcb8a4cebdae"
Oct 30 20:10:02.228266 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:10:02.228217    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:10:32.227563 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:10:32.227505    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:10:51.780870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:51.779904    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": read tcp 10.133.0.2:58446->10.133.0.153:15021: read: connection reset by peer" start-of-body=
Oct 30 20:10:51.780870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:51.780015    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": read tcp 10.133.0.2:58446->10.133.0.153:15021: read: connection reset by peer"
Oct 30 20:10:52.285655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:52.285615    4519 generic.go:356] "Generic (PLEG): container finished" podID="ccb4412e-9702-4277-bb1c-3f658f977837" containerID="817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f" exitCode=0
Oct 30 20:10:52.285873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:52.285697    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" event={"ID":"ccb4412e-9702-4277-bb1c-3f658f977837","Type":"ContainerDied","Data":"817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f"}
Oct 30 20:10:52.286414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:52.286390    4519 scope.go:122] "RemoveContainer" containerID="817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f"
Oct 30 20:10:52.289324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:52.289269    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:10:52.289478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:52.289389    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:10:52.289478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:52.289457    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:10:53.304043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:53.303997    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" event={"ID":"ccb4412e-9702-4277-bb1c-3f658f977837","Type":"ContainerStarted","Data":"271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056"}
Oct 30 20:10:53.468493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:53.468017    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 20:10:53.468493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:53.468174    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:10:53.468493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:53.468224    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 20:10:54.472417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:54.471961    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:10:54.472417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:54.472031    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 20:10:55.463493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:55.462934    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:10:55.463493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:55.463011    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 20:10:56.465075 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:56.465020    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:10:56.465833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:56.465790    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 20:10:57.462487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:57.462416    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:10:57.462828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:57.462792    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 20:10:58.465663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:58.465618    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:10:58.466392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:58.466358    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 20:10:59.468361 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:59.468317    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 20:10:59.468832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:59.468792    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 20:10:59.469751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:10:59.469725    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 20:11:02.225914 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:11:02.225868    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:11:05.251274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.251220    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"]
Oct 30 20:11:05.252327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252297    4519 cpu_manager.go:414] "containerMap: removing container" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy"
Oct 30 20:11:05.252429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252330    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy"
Oct 30 20:11:05.252429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252365    4519 cpu_manager.go:414] "containerMap: removing container" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy"
Oct 30 20:11:05.252429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252373    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy"
Oct 30 20:11:05.252429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252424    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy"
Oct 30 20:11:05.252660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252434    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy"
Oct 30 20:11:05.252660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252485    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy"
Oct 30 20:11:05.252660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252493    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy"
Oct 30 20:11:05.252826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252666    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy"
Oct 30 20:11:05.252826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252680    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b791e0b1-579c-4491-8226-418ad0459043" containerName="istio-proxy"
Oct 30 20:11:05.252826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252697    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5378b4fd-98d4-4fa8-88c3-31b19c705b45" containerName="istio-proxy"
Oct 30 20:11:05.252826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.252716    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a9878-2bb1-444d-a865-9149b3061fd1" containerName="istio-proxy"
Oct 30 20:11:05.257632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.257595    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.278455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.278414    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"]
Oct 30 20:11:05.281611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.281547    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:05.281808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.281737    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:11:05.321389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.321346    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhtd\" (UniqueName: \"kubernetes.io/projected/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-kube-api-access-fjhtd\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.321759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.321731    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.322079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.322063    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.423511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.423421    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.423827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.423804    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhtd\" (UniqueName: \"kubernetes.io/projected/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-kube-api-access-fjhtd\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.423893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.423864    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.424306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.424279    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.426644 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.426619    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.443720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.443684    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhtd\" (UniqueName: \"kubernetes.io/projected/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-kube-api-access-fjhtd\") pod \"maas-api-revoke-sub-e136ff9b-rkslw\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.585197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.585083    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:05.921346 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:11:05.921309    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0df3be_dcb7_4f1b_b15a_c1fd00de5096.slice/crio-4b5a771658f9e8bd1c26326f7935b085cb5db46dd4a6eb08e494ee1f1d80f589 WatchSource:0}: Error finding container 4b5a771658f9e8bd1c26326f7935b085cb5db46dd4a6eb08e494ee1f1d80f589: Status 404 returned error can't find the container with id 4b5a771658f9e8bd1c26326f7935b085cb5db46dd4a6eb08e494ee1f1d80f589
Oct 30 20:11:05.922435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:05.922406    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"]
Oct 30 20:11:06.470719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:06.470674    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw" event={"ID":"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096","Type":"ContainerStarted","Data":"4b5a771658f9e8bd1c26326f7935b085cb5db46dd4a6eb08e494ee1f1d80f589"}
Oct 30 20:11:07.485623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:07.485569    4519 generic.go:356] "Generic (PLEG): container finished" podID="5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" containerID="53146a4809f631b86dd944b07743b936c6ef5e3a937ac07ea3625929477d59af" exitCode=0
Oct 30 20:11:07.486231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:07.485666    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw" event={"ID":"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096","Type":"ContainerDied","Data":"53146a4809f631b86dd944b07743b936c6ef5e3a937ac07ea3625929477d59af"}
Oct 30 20:11:08.799910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.799875    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:08.936784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.936745    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-kube-api-access-fjhtd\" (UniqueName: \"kubernetes.io/projected/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-kube-api-access-fjhtd\") pod \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") "
Oct 30 20:11:08.936986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.936935    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-openshift-ca\") pod \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") "
Oct 30 20:11:08.936986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.936968    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-xks-ca\") pod \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\" (UID: \"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096\") "
Oct 30 20:11:08.937375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.937346    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-openshift-ca" pod "5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" (UID: "5d0df3be-dcb7-4f1b-b15a-c1fd00de5096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:08.937749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.937728    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:08.939176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.939145    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-xks-ca" pod "5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" (UID: "5d0df3be-dcb7-4f1b-b15a-c1fd00de5096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:08.939176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:08.939160    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-kube-api-access-fjhtd" pod "5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" (UID: "5d0df3be-dcb7-4f1b-b15a-c1fd00de5096"). InnerVolumeSpecName "kube-api-access-fjhtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:09.039154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:09.039056    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjhtd\" (UniqueName: \"kubernetes.io/projected/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-kube-api-access-fjhtd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:09.039154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:09.039094    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:09.508921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:09.508889    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"
Oct 30 20:11:09.509213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:09.508912    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw" event={"ID":"5d0df3be-dcb7-4f1b-b15a-c1fd00de5096","Type":"ContainerDied","Data":"4b5a771658f9e8bd1c26326f7935b085cb5db46dd4a6eb08e494ee1f1d80f589"}
Oct 30 20:11:09.509404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:09.509256    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5a771658f9e8bd1c26326f7935b085cb5db46dd4a6eb08e494ee1f1d80f589"
Oct 30 20:11:16.250293 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.250245    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"]
Oct 30 20:11:16.956734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.956695    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"]
Oct 30 20:11:16.957432 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.957414    4519 cpu_manager.go:414] "containerMap: removing container" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy"
Oct 30 20:11:16.957504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.957434    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy"
Oct 30 20:11:16.957549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.957504    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" containerName="revoke-keys"
Oct 30 20:11:16.957549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.957514    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" containerName="revoke-keys"
Oct 30 20:11:16.957669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.957659    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" containerName="revoke-keys"
Oct 30 20:11:16.957710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.957673    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cf99a9-cef2-4e0b-8455-47bbd2c33edf" containerName="istio-proxy"
Oct 30 20:11:16.961203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.961178    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:16.984423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.984382    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"]
Oct 30 20:11:16.988559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.988525    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:16.988736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:16.988604    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:11:17.049295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.049251    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rb5\" (UniqueName: \"kubernetes.io/projected/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-kube-api-access-g5rb5\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.049790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.049762    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.050095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.050075    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.151446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.151383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rb5\" (UniqueName: \"kubernetes.io/projected/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-kube-api-access-g5rb5\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.151911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.151884    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.152083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.152066    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.152966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.152938    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.154588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.154564    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.181148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.181112    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rb5\" (UniqueName: \"kubernetes.io/projected/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-kube-api-access-g5rb5\") pod \"maas-api-revoke-sub-ee823fe0-sdbdv\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.279414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.279318    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:17.607530 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:11:17.607356    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaf708b_2ec3_45e3_bd61_164cbb491ac8.slice/crio-d5d2353b7acc42c58b6c06d96c377ec1d2a196b3caf058448efe20835be3efa9 WatchSource:0}: Error finding container d5d2353b7acc42c58b6c06d96c377ec1d2a196b3caf058448efe20835be3efa9: Status 404 returned error can't find the container with id d5d2353b7acc42c58b6c06d96c377ec1d2a196b3caf058448efe20835be3efa9
Oct 30 20:11:17.616069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:17.616042    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"]
Oct 30 20:11:18.639211 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:18.639160    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv" event={"ID":"4eaf708b-2ec3-45e3-bd61-164cbb491ac8","Type":"ContainerStarted","Data":"dde2d9e263258b98f55487df3f5ac49847f22047818aedd1b6302b1ea6ff2476"}
Oct 30 20:11:18.639868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:18.639280    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv" event={"ID":"4eaf708b-2ec3-45e3-bd61-164cbb491ac8","Type":"ContainerStarted","Data":"d5d2353b7acc42c58b6c06d96c377ec1d2a196b3caf058448efe20835be3efa9"}
Oct 30 20:11:18.685204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:18.685092    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv" podStartSLOduration=2.685063004 podStartE2EDuration="2.685063004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:18.680191907 +0000 UTC m=+2203.180895632" watchObservedRunningTime="2026-10-30 20:11:18.685063004 +0000 UTC m=+2203.185766735"
Oct 30 20:11:19.674939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:19.671171    4519 generic.go:356] "Generic (PLEG): container finished" podID="4eaf708b-2ec3-45e3-bd61-164cbb491ac8" containerID="dde2d9e263258b98f55487df3f5ac49847f22047818aedd1b6302b1ea6ff2476" exitCode=0
Oct 30 20:11:19.674939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:19.671416    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv" event={"ID":"4eaf708b-2ec3-45e3-bd61-164cbb491ac8","Type":"ContainerDied","Data":"dde2d9e263258b98f55487df3f5ac49847f22047818aedd1b6302b1ea6ff2476"}
Oct 30 20:11:21.189654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.189615    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:21.273316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.273074    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-kube-api-access-g5rb5\" (UniqueName: \"kubernetes.io/projected/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-kube-api-access-g5rb5\") pod \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") "
Oct 30 20:11:21.273591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.273365    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-xks-ca\") pod \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") "
Oct 30 20:11:21.273591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.273542    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-openshift-ca\") pod \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\" (UID: \"4eaf708b-2ec3-45e3-bd61-164cbb491ac8\") "
Oct 30 20:11:21.282260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.282038    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-openshift-ca" pod "4eaf708b-2ec3-45e3-bd61-164cbb491ac8" (UID: "4eaf708b-2ec3-45e3-bd61-164cbb491ac8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:21.295816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.295723    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-xks-ca" pod "4eaf708b-2ec3-45e3-bd61-164cbb491ac8" (UID: "4eaf708b-2ec3-45e3-bd61-164cbb491ac8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:21.302344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.302308    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-kube-api-access-g5rb5" pod "4eaf708b-2ec3-45e3-bd61-164cbb491ac8" (UID: "4eaf708b-2ec3-45e3-bd61-164cbb491ac8"). InnerVolumeSpecName "kube-api-access-g5rb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:21.383524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.381022    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:21.383524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.381071    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rb5\" (UniqueName: \"kubernetes.io/projected/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-kube-api-access-g5rb5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:21.383524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.381089    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf708b-2ec3-45e3-bd61-164cbb491ac8-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:21.774967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.774931    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"
Oct 30 20:11:21.774967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.774957    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv" event={"ID":"4eaf708b-2ec3-45e3-bd61-164cbb491ac8","Type":"ContainerDied","Data":"d5d2353b7acc42c58b6c06d96c377ec1d2a196b3caf058448efe20835be3efa9"}
Oct 30 20:11:21.775205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:21.775015    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5d2353b7acc42c58b6c06d96c377ec1d2a196b3caf058448efe20835be3efa9"
Oct 30 20:11:31.285072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.284978    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerName="main" containerID="cri-o://69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6" gracePeriod=45
Oct 30 20:11:31.609778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.609636    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:11:31.698298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698255    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-model-cache\") pod \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") "
Oct 30 20:11:31.698542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698348    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-dshm\") pod \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") "
Oct 30 20:11:31.698542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698513    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcb333b-2caf-4b6c-878d-dd782c5a0210-kube-api-access-4t2gw\" (UniqueName: \"kubernetes.io/projected/cbcb333b-2caf-4b6c-878d-dd782c5a0210-kube-api-access-4t2gw\") pod \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") "
Oct 30 20:11:31.698695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698573    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-model-cache" pod "cbcb333b-2caf-4b6c-878d-dd782c5a0210" (UID: "cbcb333b-2caf-4b6c-878d-dd782c5a0210"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:31.698695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698642    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tmp-dir\") pod \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") "
Oct 30 20:11:31.698821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698738    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tls-certs\") pod \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") "
Oct 30 20:11:31.698821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698763    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-home\") pod \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\" (UID: \"cbcb333b-2caf-4b6c-878d-dd782c5a0210\") "
Oct 30 20:11:31.698934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.698872    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tmp-dir" pod "cbcb333b-2caf-4b6c-878d-dd782c5a0210" (UID: "cbcb333b-2caf-4b6c-878d-dd782c5a0210"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:31.699147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.699128    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-home" pod "cbcb333b-2caf-4b6c-878d-dd782c5a0210" (UID: "cbcb333b-2caf-4b6c-878d-dd782c5a0210"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:31.699811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.699782    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:31.699811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.699812    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:31.700003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.699827    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:31.701273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.701239    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcb333b-2caf-4b6c-878d-dd782c5a0210-kube-api-access-4t2gw" pod "cbcb333b-2caf-4b6c-878d-dd782c5a0210" (UID: "cbcb333b-2caf-4b6c-878d-dd782c5a0210"). InnerVolumeSpecName "kube-api-access-4t2gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:31.701403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.701328    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tls-certs" pod "cbcb333b-2caf-4b6c-878d-dd782c5a0210" (UID: "cbcb333b-2caf-4b6c-878d-dd782c5a0210"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:31.701541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.701423    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-dshm" pod "cbcb333b-2caf-4b6c-878d-dd782c5a0210" (UID: "cbcb333b-2caf-4b6c-878d-dd782c5a0210"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:31.801680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.801630    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2gw\" (UniqueName: \"kubernetes.io/projected/cbcb333b-2caf-4b6c-878d-dd782c5a0210-kube-api-access-4t2gw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:31.801680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.801679    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb333b-2caf-4b6c-878d-dd782c5a0210-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:31.801919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.801697    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb333b-2caf-4b6c-878d-dd782c5a0210-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:31.965256 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.965199    4519 generic.go:356] "Generic (PLEG): container finished" podID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerID="69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6" exitCode=0
Oct 30 20:11:31.965256 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.965244    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"
Oct 30 20:11:31.965570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.965288    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" event={"ID":"cbcb333b-2caf-4b6c-878d-dd782c5a0210","Type":"ContainerDied","Data":"69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6"}
Oct 30 20:11:31.965570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.965345    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9" event={"ID":"cbcb333b-2caf-4b6c-878d-dd782c5a0210","Type":"ContainerDied","Data":"c5bba99c82937b2c740b9e1d44bd50da0143c354093270a6d552e94185395309"}
Oct 30 20:11:31.965570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.965372    4519 scope.go:122] "RemoveContainer" containerID="69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6"
Oct 30 20:11:31.981898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.981867    4519 scope.go:122] "RemoveContainer" containerID="69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6"
Oct 30 20:11:31.982320 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:11:31.982293    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6\": container with ID starting with 69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6 not found: ID does not exist" containerID="69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6"
Oct 30 20:11:31.982436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:31.982334    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6"} err="failed to get container status \"69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6\": rpc error: code = NotFound desc = could not find container \"69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6\": container with ID starting with 69d49e1ff79a413a16403bafa7d3c72cb6c4f8996d0890c9bee180a9347baaf6 not found: ID does not exist"
Oct 30 20:11:32.020411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:32.020364    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"]
Oct 30 20:11:32.031081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:32.031044    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ef132f/test-model-ef132f-kserve-68f9588bd9-vzxz9"]
Oct 30 20:11:32.226138 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:11:32.226025    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:11:33.615173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:33.615125    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" path="/var/lib/kubelet/pods/cbcb333b-2caf-4b6c-878d-dd782c5a0210/volumes"
Oct 30 20:11:35.029636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:35.029596    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"]
Oct 30 20:11:35.041223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:35.041189    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c74a4b-d47wr"]
Oct 30 20:11:35.615352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:35.615313    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fafaa7-b169-4bee-935e-fcf1d219fefc" path="/var/lib/kubelet/pods/66fafaa7-b169-4bee-935e-fcf1d219fefc/volumes"
Oct 30 20:11:46.489645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:46.489600    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"]
Oct 30 20:11:46.490171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:46.489846    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy" containerID="cri-o://d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742" gracePeriod=30
Oct 30 20:11:51.348566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.344571    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"]
Oct 30 20:11:51.908516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.908361    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"]
Oct 30 20:11:51.909420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.909393    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerName="main"
Oct 30 20:11:51.909420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.909416    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerName="main"
Oct 30 20:11:51.909701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.909448    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4eaf708b-2ec3-45e3-bd61-164cbb491ac8" containerName="revoke-keys"
Oct 30 20:11:51.909701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.909455    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eaf708b-2ec3-45e3-bd61-164cbb491ac8" containerName="revoke-keys"
Oct 30 20:11:51.909701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.909644    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eaf708b-2ec3-45e3-bd61-164cbb491ac8" containerName="revoke-keys"
Oct 30 20:11:51.909701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.909667    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcb333b-2caf-4b6c-878d-dd782c5a0210" containerName="main"
Oct 30 20:11:51.915837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.915807    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:51.940648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.940485    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"]
Oct 30 20:11:51.941198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.940870    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:11:51.941198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:51.940932    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:52.006774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.006241    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.006774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.006418    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cpj\" (UniqueName: \"kubernetes.io/projected/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-kube-api-access-n5cpj\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.006774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.006645    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.033730 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.033701    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 20:11:52.108016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.107979    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-data\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108093    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-podinfo\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108136    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-envoy\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108188    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istiod-ca-cert\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108384    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-credential-socket\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108519    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-kube-api-access-6cv6s\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-kube-api-access-6cv6s\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108618    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-certs\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108679    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-socket\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.108775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.108726    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-token\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-token\") pod \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\" (UID: \"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674\") "
Oct 30 20:11:52.109053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.109024    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-data" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:52.109251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.109091    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cpj\" (UniqueName: \"kubernetes.io/projected/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-kube-api-access-n5cpj\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.109669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.109640    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istiod-ca-cert" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:52.109855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.109835    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.110201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.110185    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.110406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.110393    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.110529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.110516    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.110628 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.109864    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-certs" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:52.110723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.109943    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-socket" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:52.110814 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.110073    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-credential-socket" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:52.110978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.110953    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.113378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.113350    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-kube-api-access-6cv6s" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "kube-api-access-6cv6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:52.116103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.115854    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-envoy" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:52.117979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.117925    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.123587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.118877    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-podinfo" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:52.124016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.123996    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-token" pod "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" (UID: "ae5f5b1e-6a46-4dc9-bd0e-839f710e3674"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:52.129310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.129284    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cpj\" (UniqueName: \"kubernetes.io/projected/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-kube-api-access-n5cpj\") pod \"maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.213353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.213311    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cv6s\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-kube-api-access-6cv6s\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.213353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.213348    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.213353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.213359    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.213353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.213368    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.213353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.213378    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.213891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.213390    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.213891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.213403    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:52.250830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.250780    4519 generic.go:356] "Generic (PLEG): container finished" podID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerID="d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742" exitCode=0
Oct 30 20:11:52.251364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.250848    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"
Oct 30 20:11:52.251364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.250985    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" event={"ID":"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674","Type":"ContainerDied","Data":"d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742"}
Oct 30 20:11:52.251364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.251049    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8" event={"ID":"ae5f5b1e-6a46-4dc9-bd0e-839f710e3674","Type":"ContainerDied","Data":"18bdacedb97939ac5756d1d2a8385678e3dba59a0987eda46e655aa178469344"}
Oct 30 20:11:52.251364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.251079    4519 scope.go:122] "RemoveContainer" containerID="d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742"
Oct 30 20:11:52.266115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.266087    4519 scope.go:122] "RemoveContainer" containerID="a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b"
Oct 30 20:11:52.275675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.275638    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:52.282287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.282263    4519 scope.go:122] "RemoveContainer" containerID="d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742"
Oct 30 20:11:52.282832 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:11:52.282806    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742\": container with ID starting with d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742 not found: ID does not exist" containerID="d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742"
Oct 30 20:11:52.282958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.282845    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742"} err="failed to get container status \"d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742\": rpc error: code = NotFound desc = could not find container \"d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742\": container with ID starting with d861cc3f267fb1024c6288c62ebf28595c3dc8f17fd5b0703c5759b2d34d7742 not found: ID does not exist"
Oct 30 20:11:52.282958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.282873    4519 scope.go:122] "RemoveContainer" containerID="a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b"
Oct 30 20:11:52.283220 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:11:52.283199    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b\": container with ID starting with a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b not found: ID does not exist" containerID="a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b"
Oct 30 20:11:52.283296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.283233    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b"} err="failed to get container status \"a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b\": rpc error: code = NotFound desc = could not find container \"a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b\": container with ID starting with a384f1195a66576890adebefeda89e116b5fa479f71acddb1fc710c8679b7f4b not found: ID does not exist"
Oct 30 20:11:52.314614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.314578    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"]
Oct 30 20:11:52.324252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.324211    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-6rmc8"]
Oct 30 20:11:52.606791 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:11:52.606741    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84b853e_b0b7_4ec1_a9b2_d0c2ac2eff28.slice/crio-e0e79f511eaef4b2eba08c0b7e44d1bee6480ca07eb82a7c3859f7cca4baab3c WatchSource:0}: Error finding container e0e79f511eaef4b2eba08c0b7e44d1bee6480ca07eb82a7c3859f7cca4baab3c: Status 404 returned error can't find the container with id e0e79f511eaef4b2eba08c0b7e44d1bee6480ca07eb82a7c3859f7cca4baab3c
Oct 30 20:11:52.608519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:52.608452    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"]
Oct 30 20:11:53.263292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:53.263165    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh" event={"ID":"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28","Type":"ContainerStarted","Data":"b714d2e748333fa0c9dfafa169900daec80291e06977ec810c413f939c382ac9"}
Oct 30 20:11:53.263292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:53.263236    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh" event={"ID":"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28","Type":"ContainerStarted","Data":"e0e79f511eaef4b2eba08c0b7e44d1bee6480ca07eb82a7c3859f7cca4baab3c"}
Oct 30 20:11:53.305279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:53.305221    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh" podStartSLOduration=2.305202343 podStartE2EDuration="2.305202343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:53.300124212 +0000 UTC m=+2237.800827938" watchObservedRunningTime="2026-10-30 20:11:53.305202343 +0000 UTC m=+2237.805906107"
Oct 30 20:11:53.617030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:53.616938    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" path="/var/lib/kubelet/pods/ae5f5b1e-6a46-4dc9-bd0e-839f710e3674/volumes"
Oct 30 20:11:54.045850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:54.045819    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"]
Oct 30 20:11:54.076730 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:54.076691    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245312d4-q6ck5"]
Oct 30 20:11:54.275040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:54.274998    4519 generic.go:356] "Generic (PLEG): container finished" podID="a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" containerID="b714d2e748333fa0c9dfafa169900daec80291e06977ec810c413f939c382ac9" exitCode=0
Oct 30 20:11:54.275239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:54.275084    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh" event={"ID":"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28","Type":"ContainerDied","Data":"b714d2e748333fa0c9dfafa169900daec80291e06977ec810c413f939c382ac9"}
Oct 30 20:11:55.489679 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.489649    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:11:55.582290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.582150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-kube-api-access-n5cpj\" (UniqueName: \"kubernetes.io/projected/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-kube-api-access-n5cpj\") pod \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") "
Oct 30 20:11:55.582523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.582401    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-openshift-ca\") pod \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") "
Oct 30 20:11:55.582638 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.582550    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-xks-ca\") pod \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\" (UID: \"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28\") "
Oct 30 20:11:55.583177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.583138    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-openshift-ca" pod "a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" (UID: "a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:55.583543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.583513    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:55.585162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.585126    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-kube-api-access-n5cpj" pod "a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" (UID: "a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28"). InnerVolumeSpecName "kube-api-access-n5cpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:55.585662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.585639    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-xks-ca" pod "a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" (UID: "a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:55.614410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.614379    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47dc1ed1-71ff-46e4-882c-a6776b0eae15" path="/var/lib/kubelet/pods/47dc1ed1-71ff-46e4-882c-a6776b0eae15/volumes"
Oct 30 20:11:55.684920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.684881    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5cpj\" (UniqueName: \"kubernetes.io/projected/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-kube-api-access-n5cpj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:55.684920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:55.684914    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:11:56.298253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:56.298206    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh" event={"ID":"a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28","Type":"ContainerDied","Data":"e0e79f511eaef4b2eba08c0b7e44d1bee6480ca07eb82a7c3859f7cca4baab3c"}
Oct 30 20:11:56.298448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:56.298278    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e79f511eaef4b2eba08c0b7e44d1bee6480ca07eb82a7c3859f7cca4baab3c"
Oct 30 20:11:56.298448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:11:56.298353    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"
Oct 30 20:12:02.231461 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:12:02.231417    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:12:06.376753 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.376705    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh" podUID="9de8bd51-bab6-409e-aa03-9a40668303a9" containerName="main" containerID="cri-o://1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e" gracePeriod=45
Oct 30 20:12:06.700292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.700258    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:12:06.733776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.733740    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de8bd51-bab6-409e-aa03-9a40668303a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9de8bd51-bab6-409e-aa03-9a40668303a9-tls-certs\") pod \"9de8bd51-bab6-409e-aa03-9a40668303a9\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") "
Oct 30 20:12:06.733958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.733883    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de8bd51-bab6-409e-aa03-9a40668303a9-kube-api-access-rf82f\" (UniqueName: \"kubernetes.io/projected/9de8bd51-bab6-409e-aa03-9a40668303a9-kube-api-access-rf82f\") pod \"9de8bd51-bab6-409e-aa03-9a40668303a9\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") "
Oct 30 20:12:06.734127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.734111    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-home\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-home\") pod \"9de8bd51-bab6-409e-aa03-9a40668303a9\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") "
Oct 30 20:12:06.734236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.734177    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-model-cache\") pod \"9de8bd51-bab6-409e-aa03-9a40668303a9\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") "
Oct 30 20:12:06.734236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.734209    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-tmp-dir\") pod \"9de8bd51-bab6-409e-aa03-9a40668303a9\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") "
Oct 30 20:12:06.734370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.734342    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-dshm\") pod \"9de8bd51-bab6-409e-aa03-9a40668303a9\" (UID: \"9de8bd51-bab6-409e-aa03-9a40668303a9\") "
Oct 30 20:12:06.734684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.734522    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-home" pod "9de8bd51-bab6-409e-aa03-9a40668303a9" (UID: "9de8bd51-bab6-409e-aa03-9a40668303a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:06.734684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.734640    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-model-cache" pod "9de8bd51-bab6-409e-aa03-9a40668303a9" (UID: "9de8bd51-bab6-409e-aa03-9a40668303a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:06.734684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.734650    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-tmp-dir" pod "9de8bd51-bab6-409e-aa03-9a40668303a9" (UID: "9de8bd51-bab6-409e-aa03-9a40668303a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:06.735277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.735252    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:06.735366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.735282    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:06.735366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.735300    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:06.736347 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.736322    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de8bd51-bab6-409e-aa03-9a40668303a9-tls-certs" pod "9de8bd51-bab6-409e-aa03-9a40668303a9" (UID: "9de8bd51-bab6-409e-aa03-9a40668303a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:06.736493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.736423    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de8bd51-bab6-409e-aa03-9a40668303a9-kube-api-access-rf82f" pod "9de8bd51-bab6-409e-aa03-9a40668303a9" (UID: "9de8bd51-bab6-409e-aa03-9a40668303a9"). InnerVolumeSpecName "kube-api-access-rf82f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:06.736576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.736555    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-dshm" pod "9de8bd51-bab6-409e-aa03-9a40668303a9" (UID: "9de8bd51-bab6-409e-aa03-9a40668303a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:06.837143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.837109    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf82f\" (UniqueName: \"kubernetes.io/projected/9de8bd51-bab6-409e-aa03-9a40668303a9-kube-api-access-rf82f\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:06.837143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.837140    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de8bd51-bab6-409e-aa03-9a40668303a9-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:06.837143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:06.837152    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de8bd51-bab6-409e-aa03-9a40668303a9-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:07.050437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.050390    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"]
Oct 30 20:12:07.050759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.050738    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1d6b1a-5xr2w"]
Oct 30 20:12:07.410204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.410102    4519 generic.go:356] "Generic (PLEG): container finished" podID="9de8bd51-bab6-409e-aa03-9a40668303a9" containerID="1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e" exitCode=0
Oct 30 20:12:07.410204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.410179    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"
Oct 30 20:12:07.410789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.410181    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh" event={"ID":"9de8bd51-bab6-409e-aa03-9a40668303a9","Type":"ContainerDied","Data":"1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e"}
Oct 30 20:12:07.410789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.410312    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh" event={"ID":"9de8bd51-bab6-409e-aa03-9a40668303a9","Type":"ContainerDied","Data":"29f8b02b5b927335dd8b5990a4b6d65840ba254f907832a297002f76db9c3a9c"}
Oct 30 20:12:07.410789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.410346    4519 scope.go:122] "RemoveContainer" containerID="1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e"
Oct 30 20:12:07.425956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.425928    4519 scope.go:122] "RemoveContainer" containerID="1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e"
Oct 30 20:12:07.426459 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:12:07.426415    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e\": container with ID starting with 1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e not found: ID does not exist" containerID="1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e"
Oct 30 20:12:07.426622 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.426460    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e"} err="failed to get container status \"1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e\": rpc error: code = NotFound desc = could not find container \"1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e\": container with ID starting with 1bb1433b583c97adce2dc3a764a5322a7dc7dfb18b4c69ff346353312cf3490e not found: ID does not exist"
Oct 30 20:12:07.466407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.466370    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"]
Oct 30 20:12:07.475513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.475461    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-936f70/test-model-936f70-kserve-85fc5647d5-69cwh"]
Oct 30 20:12:07.614087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.614048    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283434f1-16fa-42d1-a988-2443df18643a" path="/var/lib/kubelet/pods/283434f1-16fa-42d1-a988-2443df18643a/volumes"
Oct 30 20:12:07.614652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:07.614636    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de8bd51-bab6-409e-aa03-9a40668303a9" path="/var/lib/kubelet/pods/9de8bd51-bab6-409e-aa03-9a40668303a9/volumes"
Oct 30 20:12:17.198648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:17.198612    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"]
Oct 30 20:12:17.199098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:17.198818    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy" containerID="cri-o://0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a" gracePeriod=30
Oct 30 20:12:22.581596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.581561    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:12:22.683944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.683899    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-podinfo\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.683963    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-envoy\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.684029    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-certs\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.684076    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-kube-api-access-bsvj8\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.684188    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-token\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-token\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.684242    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-credential-socket\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.684287    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157b060d-29e9-4e2d-bf09-6f9d9b177082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b060d-29e9-4e2d-bf09-6f9d9b177082-istiod-ca-cert\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.684355    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-data\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.684402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.684380    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-socket\") pod \"157b060d-29e9-4e2d-bf09-6f9d9b177082\" (UID: \"157b060d-29e9-4e2d-bf09-6f9d9b177082\") "
Oct 30 20:12:22.685434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.685230    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-socket" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.686533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.685893    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-certs" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.687776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.687488    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-data" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.687776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.687546    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-credential-socket" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.687776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.687739    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157b060d-29e9-4e2d-bf09-6f9d9b177082-istiod-ca-cert" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:22.696450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.696409    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-token" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:22.698456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.697896    4519 generic.go:356] "Generic (PLEG): container finished" podID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerID="0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a" exitCode=0
Oct 30 20:12:22.698456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.697959    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"
Oct 30 20:12:22.698456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.698210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" event={"ID":"157b060d-29e9-4e2d-bf09-6f9d9b177082","Type":"ContainerDied","Data":"0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a"}
Oct 30 20:12:22.698456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.698266    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm" event={"ID":"157b060d-29e9-4e2d-bf09-6f9d9b177082","Type":"ContainerDied","Data":"27afc64d66b00546191f058c15b87947bf17dbb2aae3c25423752fad297e33bc"}
Oct 30 20:12:22.698456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.698299    4519 scope.go:122] "RemoveContainer" containerID="0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a"
Oct 30 20:12:22.699694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.699616    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-kube-api-access-bsvj8" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "kube-api-access-bsvj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:22.701724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.701683    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-envoy" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.707988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.707953    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-podinfo" pod "157b060d-29e9-4e2d-bf09-6f9d9b177082" (UID: "157b060d-29e9-4e2d-bf09-6f9d9b177082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:22.733623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.733586    4519 scope.go:122] "RemoveContainer" containerID="0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a"
Oct 30 20:12:22.734026 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:12:22.733998    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a\": container with ID starting with 0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a not found: ID does not exist" containerID="0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a"
Oct 30 20:12:22.734138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.734052    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a"} err="failed to get container status \"0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a\": rpc error: code = NotFound desc = could not find container \"0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a\": container with ID starting with 0457b3ba63b16450fd9539586079998ad2c8c9622d0bc714e72076d67137410a not found: ID does not exist"
Oct 30 20:12:22.786787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786745    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.786787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786788    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.787017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786803    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b060d-29e9-4e2d-bf09-6f9d9b177082-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.787017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786816    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.787017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786830    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.787017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786841    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.787017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786854    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.787017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786866    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b060d-29e9-4e2d-bf09-6f9d9b177082-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:22.787017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:22.786878    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/157b060d-29e9-4e2d-bf09-6f9d9b177082-kube-api-access-bsvj8\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:12:23.051150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:23.051115    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"]
Oct 30 20:12:23.063249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:23.063207    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-hfvxm"]
Oct 30 20:12:23.617671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:23.617633    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" path="/var/lib/kubelet/pods/157b060d-29e9-4e2d-bf09-6f9d9b177082/volumes"
Oct 30 20:12:32.221591 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:12:32.221549    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:12:43.757360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:43.756715    4519 scope.go:122] "RemoveContainer" containerID="e9f3a4142fade18c30f276f3f601bb7de3f195bf008721957c207dc467a9e62c"
Oct 30 20:12:53.474774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:53.474676    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"]
Oct 30 20:12:53.478952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:53.478921    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef132f-jsnnh"]
Oct 30 20:12:53.638893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:12:53.638852    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" path="/var/lib/kubelet/pods/a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28/volumes"
Oct 30 20:13:00.740708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:00.740671    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"]
Oct 30 20:13:00.741288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:00.740954    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" containerID="cri-o://4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae" gracePeriod=30
Oct 30 20:13:01.291618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.291573    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"]
Oct 30 20:13:01.292260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292243    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9de8bd51-bab6-409e-aa03-9a40668303a9" containerName="main"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292260    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de8bd51-bab6-409e-aa03-9a40668303a9" containerName="main"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292274    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" containerName="revoke-keys"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292279    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" containerName="revoke-keys"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292291    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292296    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292311    4519 cpu_manager.go:414] "containerMap: removing container" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292316    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292330    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy"
Oct 30 20:13:01.292338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292335    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy"
Oct 30 20:13:01.292850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292425    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de8bd51-bab6-409e-aa03-9a40668303a9" containerName="main"
Oct 30 20:13:01.292850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292434    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84b853e-b0b7-4ec1-a9b2-d0c2ac2eff28" containerName="revoke-keys"
Oct 30 20:13:01.292850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292442    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy"
Oct 30 20:13:01.292850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.292449    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="157b060d-29e9-4e2d-bf09-6f9d9b177082" containerName="istio-proxy"
Oct 30 20:13:01.297656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.297632    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.315763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.315731    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"]
Oct 30 20:13:01.351039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.351004    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.351450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.351427    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdrr2\" (UniqueName: \"kubernetes.io/projected/19147efd-9c8f-4feb-981b-99dbedbe7d40-kube-api-access-bdrr2\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.351814 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.351792    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.370783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.370758    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:01.370901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.370826    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:13:01.391137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.391107    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:13:01.391246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.391165    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:13:01.454172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.454125    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.454400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.454286    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.454490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.454428    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdrr2\" (UniqueName: \"kubernetes.io/projected/19147efd-9c8f-4feb-981b-99dbedbe7d40-kube-api-access-bdrr2\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.455226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.455196    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.456816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.456788    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.473163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.473135    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdrr2\" (UniqueName: \"kubernetes.io/projected/19147efd-9c8f-4feb-981b-99dbedbe7d40-kube-api-access-bdrr2\") pod \"maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.611648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.611564    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:01.922400 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:13:01.922365    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19147efd_9c8f_4feb_981b_99dbedbe7d40.slice/crio-0bc46a920b278c6f1a8fe12e1e998181174efc43a20f59c45cdabf6bf280100d WatchSource:0}: Error finding container 0bc46a920b278c6f1a8fe12e1e998181174efc43a20f59c45cdabf6bf280100d: Status 404 returned error can't find the container with id 0bc46a920b278c6f1a8fe12e1e998181174efc43a20f59c45cdabf6bf280100d
Oct 30 20:13:01.923729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:01.923703    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"]
Oct 30 20:13:02.222907 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:13:02.222828    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:13:02.272224 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:02.272191    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct" event={"ID":"19147efd-9c8f-4feb-981b-99dbedbe7d40","Type":"ContainerStarted","Data":"0bc46a920b278c6f1a8fe12e1e998181174efc43a20f59c45cdabf6bf280100d"}
Oct 30 20:13:03.284102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:03.284064    4519 generic.go:356] "Generic (PLEG): container finished" podID="19147efd-9c8f-4feb-981b-99dbedbe7d40" containerID="f1ddf8eac8f1ea99cac9fe7a4158bc0c6d439453bdd7e87b3851d46dfac1a17c" exitCode=0
Oct 30 20:13:03.284629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:03.284156    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct" event={"ID":"19147efd-9c8f-4feb-981b-99dbedbe7d40","Type":"ContainerDied","Data":"f1ddf8eac8f1ea99cac9fe7a4158bc0c6d439453bdd7e87b3851d46dfac1a17c"}
Oct 30 20:13:04.463753 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.463721    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:04.615068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.614965    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-openshift-ca\") pod \"19147efd-9c8f-4feb-981b-99dbedbe7d40\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") "
Oct 30 20:13:04.615261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.615068    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19147efd-9c8f-4feb-981b-99dbedbe7d40-kube-api-access-bdrr2\" (UniqueName: \"kubernetes.io/projected/19147efd-9c8f-4feb-981b-99dbedbe7d40-kube-api-access-bdrr2\") pod \"19147efd-9c8f-4feb-981b-99dbedbe7d40\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") "
Oct 30 20:13:04.615261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.615116    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-xks-ca\") pod \"19147efd-9c8f-4feb-981b-99dbedbe7d40\" (UID: \"19147efd-9c8f-4feb-981b-99dbedbe7d40\") "
Oct 30 20:13:04.615430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.615399    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-openshift-ca" pod "19147efd-9c8f-4feb-981b-99dbedbe7d40" (UID: "19147efd-9c8f-4feb-981b-99dbedbe7d40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:04.615934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.615910    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:04.624042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.624010    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19147efd-9c8f-4feb-981b-99dbedbe7d40-kube-api-access-bdrr2" pod "19147efd-9c8f-4feb-981b-99dbedbe7d40" (UID: "19147efd-9c8f-4feb-981b-99dbedbe7d40"). InnerVolumeSpecName "kube-api-access-bdrr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:04.624433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.624404    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-xks-ca" pod "19147efd-9c8f-4feb-981b-99dbedbe7d40" (UID: "19147efd-9c8f-4feb-981b-99dbedbe7d40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:04.719773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.719734    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdrr2\" (UniqueName: \"kubernetes.io/projected/19147efd-9c8f-4feb-981b-99dbedbe7d40-kube-api-access-bdrr2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:04.719773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:04.719774    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19147efd-9c8f-4feb-981b-99dbedbe7d40-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:05.302543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:05.302500    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct" event={"ID":"19147efd-9c8f-4feb-981b-99dbedbe7d40","Type":"ContainerDied","Data":"0bc46a920b278c6f1a8fe12e1e998181174efc43a20f59c45cdabf6bf280100d"}
Oct 30 20:13:05.302543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:05.302543    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"
Oct 30 20:13:05.302806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:05.302584    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bc46a920b278c6f1a8fe12e1e998181174efc43a20f59c45cdabf6bf280100d"
Oct 30 20:13:06.063741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.063710    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:13:06.139755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.139712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-envoy\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.139956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.139926    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-socket\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.140108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.139976    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-data\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.140108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.140065    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-credential-socket\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.140237 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.140118    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-kube-api-access-h96md\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-kube-api-access-h96md\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.140237 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.140150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-token\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-token\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.140237 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.140205    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-certs\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.140413 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.140259    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-podinfo\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.140413 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.140323    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istiod-ca-cert\") pod \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\" (UID: \"aa8b2862-9442-47d5-af3c-ff55bb26b17c\") "
Oct 30 20:13:06.141639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.141559    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istiod-ca-cert" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:06.142529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.142436    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-data" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:06.142776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.142736    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-socket" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:06.143095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.143067    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-certs" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:06.146727 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.146682    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-credential-socket" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:06.161103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.161074    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-envoy" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:06.161186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.161083    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-podinfo" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:06.161186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.161144    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-token" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:06.161391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.161371    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-kube-api-access-h96md" pod "aa8b2862-9442-47d5-af3c-ff55bb26b17c" (UID: "aa8b2862-9442-47d5-af3c-ff55bb26b17c"). InnerVolumeSpecName "kube-api-access-h96md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:06.242996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.242962    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.242996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.242991    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.243179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.243001    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.243179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.243013    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h96md\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-kube-api-access-h96md\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.243179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.243022    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.243179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.243030    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.243179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.243038    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.243179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.243045    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.243179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.243053    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8b2862-9442-47d5-af3c-ff55bb26b17c-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:06.341048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.340959    4519 generic.go:356] "Generic (PLEG): container finished" podID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerID="4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae" exitCode=0
Oct 30 20:13:06.341048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.341017    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"
Oct 30 20:13:06.341048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.341038    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" event={"ID":"aa8b2862-9442-47d5-af3c-ff55bb26b17c","Type":"ContainerDied","Data":"4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae"}
Oct 30 20:13:06.341321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.341092    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz" event={"ID":"aa8b2862-9442-47d5-af3c-ff55bb26b17c","Type":"ContainerDied","Data":"385b9a4c45ee48745e7bc5d8376fa485ced31ea2489c858ea03f21f11372d0b9"}
Oct 30 20:13:06.341321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.341116    4519 scope.go:122] "RemoveContainer" containerID="4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae"
Oct 30 20:13:06.353275 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.353255    4519 scope.go:122] "RemoveContainer" containerID="0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c"
Oct 30 20:13:06.363916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.363892    4519 scope.go:122] "RemoveContainer" containerID="4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae"
Oct 30 20:13:06.364218 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:13:06.364193    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae\": container with ID starting with 4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae not found: ID does not exist" containerID="4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae"
Oct 30 20:13:06.364278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.364230    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae"} err="failed to get container status \"4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae\": rpc error: code = NotFound desc = could not find container \"4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae\": container with ID starting with 4826616ffa9fb5555e87537c5662356f841db8c01e4c7254037cd26307eacdae not found: ID does not exist"
Oct 30 20:13:06.364278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.364255    4519 scope.go:122] "RemoveContainer" containerID="0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c"
Oct 30 20:13:06.364708 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:13:06.364678    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c\": container with ID starting with 0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c not found: ID does not exist" containerID="0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c"
Oct 30 20:13:06.364789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.364706    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c"} err="failed to get container status \"0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c\": rpc error: code = NotFound desc = could not find container \"0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c\": container with ID starting with 0af9fed49b03f379c40da24876213ee05f8bc387aa161b30d87b094526245f9c not found: ID does not exist"
Oct 30 20:13:06.395386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.395353    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"]
Oct 30 20:13:06.399516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:06.399489    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ef132f-openshift-default-5f9546d845-crtdz"]
Oct 30 20:13:07.613953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:07.613916    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" path="/var/lib/kubelet/pods/aa8b2862-9442-47d5-af3c-ff55bb26b17c/volumes"
Oct 30 20:13:31.803854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:31.803818    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"]
Oct 30 20:13:31.804327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:31.804019    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy" containerID="cri-o://de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df" gracePeriod=30
Oct 30 20:13:32.221711 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:13:32.221675    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:13:36.373033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.372991    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"]
Oct 30 20:13:36.373960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.373936    4519 cpu_manager.go:414] "containerMap: removing container" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy"
Oct 30 20:13:36.374053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.373961    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy"
Oct 30 20:13:36.374053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.373993    4519 cpu_manager.go:414] "containerMap: removing container" podUID="19147efd-9c8f-4feb-981b-99dbedbe7d40" containerName="revoke-keys"
Oct 30 20:13:36.374053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.374002    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19147efd-9c8f-4feb-981b-99dbedbe7d40" containerName="revoke-keys"
Oct 30 20:13:36.374053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.374031    4519 cpu_manager.go:414] "containerMap: removing container" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy"
Oct 30 20:13:36.374053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.374040    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy"
Oct 30 20:13:36.374330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.374192    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="19147efd-9c8f-4feb-981b-99dbedbe7d40" containerName="revoke-keys"
Oct 30 20:13:36.374330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.374210    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy"
Oct 30 20:13:36.374330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.374227    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8b2862-9442-47d5-af3c-ff55bb26b17c" containerName="istio-proxy"
Oct 30 20:13:36.374330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.374238    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5f5b1e-6a46-4dc9-bd0e-839f710e3674" containerName="istio-proxy"
Oct 30 20:13:36.378361 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.378335    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.395590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.395539    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"]
Oct 30 20:13:36.401019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.400994    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:36.401019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.401004    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:13:36.513189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.513156    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.513366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.513208    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6gp\" (UniqueName: \"kubernetes.io/projected/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-kube-api-access-2s6gp\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.513517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.513491    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.615345 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.615308    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.615583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.615433    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.615583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.615457    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6gp\" (UniqueName: \"kubernetes.io/projected/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-kube-api-access-2s6gp\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.616061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.616034    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.617946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.617911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.634181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.634116    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6gp\" (UniqueName: \"kubernetes.io/projected/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-kube-api-access-2s6gp\") pod \"maas-api-revoke-sub-4bd7b75c-64679\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:36.695346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:36.695312    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:37.009700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.009294    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"]
Oct 30 20:13:37.069860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.069837    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228345    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-data\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228422    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-socket\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228502    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-token\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-token\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228641    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-envoy\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228759    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-kube-api-access-hwvdv\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-kube-api-access-hwvdv\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228804    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-podinfo\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228893    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-credential-socket\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.228977    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istiod-ca-cert\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.229027    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-data" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.229084    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-certs\") pod \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\" (UID: \"2a1bce2e-135b-44ff-9cf3-0dba343d26ef\") "
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.229862    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.230135    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-certs" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:37.233110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.230258    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-socket" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:37.235459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.233337    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-credential-socket" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:37.235459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.234790    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istiod-ca-cert" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:37.248612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.248579    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-envoy" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:37.248754 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.248587    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-podinfo" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:37.248836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.248749    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-token" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:37.248895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.248840    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-kube-api-access-hwvdv" pod "2a1bce2e-135b-44ff-9cf3-0dba343d26ef" (UID: "2a1bce2e-135b-44ff-9cf3-0dba343d26ef"). InnerVolumeSpecName "kube-api-access-hwvdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:37.332370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332334    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.332370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332365    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.332370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332375    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.332665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332384    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvdv\" (UniqueName: \"kubernetes.io/projected/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-kube-api-access-hwvdv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.332665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332394    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.332665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332403    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.332665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332411    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.332665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.332420    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1bce2e-135b-44ff-9cf3-0dba343d26ef-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:37.623371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.623276    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679" event={"ID":"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d","Type":"ContainerStarted","Data":"01ca2b70b0e20f426cfc5a9b7ca4b8e2749eca0e61bcec57dfd5b5f7efc71686"}
Oct 30 20:13:37.623371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.623336    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679" event={"ID":"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d","Type":"ContainerStarted","Data":"338e65d266b851f7d79f1429c049024e3234a39f6bf58da2fc88af465b075534"}
Oct 30 20:13:37.624772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.624740    4519 generic.go:356] "Generic (PLEG): container finished" podID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerID="de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df" exitCode=0
Oct 30 20:13:37.624904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.624801    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"
Oct 30 20:13:37.624904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.624825    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" event={"ID":"2a1bce2e-135b-44ff-9cf3-0dba343d26ef","Type":"ContainerDied","Data":"de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df"}
Oct 30 20:13:37.624904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.624881    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg" event={"ID":"2a1bce2e-135b-44ff-9cf3-0dba343d26ef","Type":"ContainerDied","Data":"a7cbd0250532cd6f6a0d13d2ad0f1dcbefd655c22daf23177648d317c32fc4c1"}
Oct 30 20:13:37.624904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.624904    4519 scope.go:122] "RemoveContainer" containerID="de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df"
Oct 30 20:13:37.645750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.645731    4519 scope.go:122] "RemoveContainer" containerID="de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df"
Oct 30 20:13:37.646043 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:13:37.646025    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df\": container with ID starting with de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df not found: ID does not exist" containerID="de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df"
Oct 30 20:13:37.646122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.646051    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df"} err="failed to get container status \"de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df\": rpc error: code = NotFound desc = could not find container \"de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df\": container with ID starting with de7751dce89d98611b19b2b35c355540202984894daf4402a7f8563ca91218df not found: ID does not exist"
Oct 30 20:13:37.661765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.661423    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679" podStartSLOduration=1.661404479 podStartE2EDuration="1.661404479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:37.660673921 +0000 UTC m=+2342.161377647" watchObservedRunningTime="2026-10-30 20:13:37.661404479 +0000 UTC m=+2342.162108203"
Oct 30 20:13:37.702890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.702857    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"]
Oct 30 20:13:37.734489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:37.728880    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-hswfg"]
Oct 30 20:13:38.636447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:38.636412    4519 generic.go:356] "Generic (PLEG): container finished" podID="6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" containerID="01ca2b70b0e20f426cfc5a9b7ca4b8e2749eca0e61bcec57dfd5b5f7efc71686" exitCode=0
Oct 30 20:13:38.636870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:38.636484    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679" event={"ID":"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d","Type":"ContainerDied","Data":"01ca2b70b0e20f426cfc5a9b7ca4b8e2749eca0e61bcec57dfd5b5f7efc71686"}
Oct 30 20:13:39.615174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.615129    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" path="/var/lib/kubelet/pods/2a1bce2e-135b-44ff-9cf3-0dba343d26ef/volumes"
Oct 30 20:13:39.823704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.823674    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:39.888534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.888435    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-xks-ca\") pod \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") "
Oct 30 20:13:39.889417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.889389    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-kube-api-access-2s6gp\" (UniqueName: \"kubernetes.io/projected/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-kube-api-access-2s6gp\") pod \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") "
Oct 30 20:13:39.889871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.889690    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-openshift-ca\") pod \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\" (UID: \"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d\") "
Oct 30 20:13:39.890216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.890189    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-openshift-ca" pod "6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" (UID: "6cbc1ba5-0593-4017-a8ac-6f4808f4f89d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:39.891109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.891085    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:39.891979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.891789    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-kube-api-access-2s6gp" pod "6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" (UID: "6cbc1ba5-0593-4017-a8ac-6f4808f4f89d"). InnerVolumeSpecName "kube-api-access-2s6gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:39.892506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.892445    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-xks-ca" pod "6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" (UID: "6cbc1ba5-0593-4017-a8ac-6f4808f4f89d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:39.992523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.992491    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:39.992523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:39.992521    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6gp\" (UniqueName: \"kubernetes.io/projected/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d-kube-api-access-2s6gp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:40.659033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:40.659000    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"
Oct 30 20:13:40.659252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:40.659039    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679" event={"ID":"6cbc1ba5-0593-4017-a8ac-6f4808f4f89d","Type":"ContainerDied","Data":"338e65d266b851f7d79f1429c049024e3234a39f6bf58da2fc88af465b075534"}
Oct 30 20:13:40.659252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:40.659098    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="338e65d266b851f7d79f1429c049024e3234a39f6bf58da2fc88af465b075534"
Oct 30 20:13:42.106425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.106393    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"]
Oct 30 20:13:42.107020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.107004    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" containerName="revoke-keys"
Oct 30 20:13:42.107065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.107022    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" containerName="revoke-keys"
Oct 30 20:13:42.107065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.107037    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy"
Oct 30 20:13:42.107065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.107042    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy"
Oct 30 20:13:42.107177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.107135    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" containerName="istio-proxy"
Oct 30 20:13:42.107177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.107152    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" containerName="revoke-keys"
Oct 30 20:13:42.110804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.110787    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.129624 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.129597    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"]
Oct 30 20:13:42.135835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.135805    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:42.136011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.135852    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:13:42.224269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.224237    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.224436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.224390    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.224557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.224539    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbrf\" (UniqueName: \"kubernetes.io/projected/3b81735f-d028-4a80-85d4-57cfb011c4b2-kube-api-access-bkbrf\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.325506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.325435    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.325692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.325596    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbrf\" (UniqueName: \"kubernetes.io/projected/3b81735f-d028-4a80-85d4-57cfb011c4b2-kube-api-access-bkbrf\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.325770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.325741    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.326130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.326109    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.328185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.328165    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.344274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.344252    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbrf\" (UniqueName: \"kubernetes.io/projected/3b81735f-d028-4a80-85d4-57cfb011c4b2-kube-api-access-bkbrf\") pod \"maas-api-revoke-sub-fe359514-qdv88\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.430665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.430639    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:42.740682 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:13:42.740645    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b81735f_d028_4a80_85d4_57cfb011c4b2.slice/crio-14669acd9a4dfca2bdd9a054853c7612d63c05e1d6d6afd45f5ed8d02843f0ad WatchSource:0}: Error finding container 14669acd9a4dfca2bdd9a054853c7612d63c05e1d6d6afd45f5ed8d02843f0ad: Status 404 returned error can't find the container with id 14669acd9a4dfca2bdd9a054853c7612d63c05e1d6d6afd45f5ed8d02843f0ad
Oct 30 20:13:42.741386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:42.741359    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"]
Oct 30 20:13:43.499819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.499784    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"]
Oct 30 20:13:43.504288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.504262    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.523535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.523508    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"]
Oct 30 20:13:43.544845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.544809    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.544992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.544953    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5mw\" (UniqueName: \"kubernetes.io/projected/68dc8b8a-693f-436e-9219-f91abd6b1b94-kube-api-access-zg5mw\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.545155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.545131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.647393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.647348    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.647603 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.647584    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.647702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.647669    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5mw\" (UniqueName: \"kubernetes.io/projected/68dc8b8a-693f-436e-9219-f91abd6b1b94-kube-api-access-zg5mw\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.648677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.648653    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.650440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.650410    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.666807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.666781    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5mw\" (UniqueName: \"kubernetes.io/projected/68dc8b8a-693f-436e-9219-f91abd6b1b94-kube-api-access-zg5mw\") pod \"maas-api-revoke-sub-addf6c77-24zqs\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.687956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.687392    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88" event={"ID":"3b81735f-d028-4a80-85d4-57cfb011c4b2","Type":"ContainerStarted","Data":"94f9c4bf6dc3b45f36bd189381aaf94e0a62bc00193007b3a8ba13406de6f72b"}
Oct 30 20:13:43.688102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.687999    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88" event={"ID":"3b81735f-d028-4a80-85d4-57cfb011c4b2","Type":"ContainerStarted","Data":"14669acd9a4dfca2bdd9a054853c7612d63c05e1d6d6afd45f5ed8d02843f0ad"}
Oct 30 20:13:43.729521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.729430    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88" podStartSLOduration=1.7294164300000001 podStartE2EDuration="1.72941643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:43.727493521 +0000 UTC m=+2348.228197255" watchObservedRunningTime="2026-10-30 20:13:43.72941643 +0000 UTC m=+2348.230120154"
Oct 30 20:13:43.821979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.821948    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:43.825549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.825527    4519 scope.go:122] "RemoveContainer" containerID="ab38e5a4a1f59a07d5edfc00108284c9cd43ccded23da8d9c3f586f751f2e41d"
Oct 30 20:13:43.836853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.836828    4519 scope.go:122] "RemoveContainer" containerID="b2363595975e9f1dacba23a7475ac677558173ef52e01e32a17d772c8f49e2da"
Oct 30 20:13:43.859292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:43.859266    4519 scope.go:122] "RemoveContainer" containerID="c1f8bfab21580d670e8907e6683277f73c92b231b757543f9e2fc2a06527137a"
Oct 30 20:13:44.138886 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:13:44.138855    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dc8b8a_693f_436e_9219_f91abd6b1b94.slice/crio-87c4789ae9ec0390d59e3b03b23006168ca2b56fe38ce036a2c50e98f6e5686b WatchSource:0}: Error finding container 87c4789ae9ec0390d59e3b03b23006168ca2b56fe38ce036a2c50e98f6e5686b: Status 404 returned error can't find the container with id 87c4789ae9ec0390d59e3b03b23006168ca2b56fe38ce036a2c50e98f6e5686b
Oct 30 20:13:44.142615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:44.142587    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"]
Oct 30 20:13:44.743688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:44.743639    4519 generic.go:356] "Generic (PLEG): container finished" podID="3b81735f-d028-4a80-85d4-57cfb011c4b2" containerID="94f9c4bf6dc3b45f36bd189381aaf94e0a62bc00193007b3a8ba13406de6f72b" exitCode=0
Oct 30 20:13:44.744129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:44.743777    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88" event={"ID":"3b81735f-d028-4a80-85d4-57cfb011c4b2","Type":"ContainerDied","Data":"94f9c4bf6dc3b45f36bd189381aaf94e0a62bc00193007b3a8ba13406de6f72b"}
Oct 30 20:13:44.745801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:44.745774    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs" event={"ID":"68dc8b8a-693f-436e-9219-f91abd6b1b94","Type":"ContainerStarted","Data":"f71bac934edb82a80b63691bc6856052fd0620dadfcbca2b170467d961bf71dc"}
Oct 30 20:13:44.745936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:44.745830    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs" event={"ID":"68dc8b8a-693f-436e-9219-f91abd6b1b94","Type":"ContainerStarted","Data":"87c4789ae9ec0390d59e3b03b23006168ca2b56fe38ce036a2c50e98f6e5686b"}
Oct 30 20:13:44.823397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:44.823289    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs" podStartSLOduration=1.823270749 podStartE2EDuration="1.823270749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:44.820887812 +0000 UTC m=+2349.321591619" watchObservedRunningTime="2026-10-30 20:13:44.823270749 +0000 UTC m=+2349.323974485"
Oct 30 20:13:45.756404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:45.756369    4519 generic.go:356] "Generic (PLEG): container finished" podID="68dc8b8a-693f-436e-9219-f91abd6b1b94" containerID="f71bac934edb82a80b63691bc6856052fd0620dadfcbca2b170467d961bf71dc" exitCode=0
Oct 30 20:13:45.756882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:45.756448    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs" event={"ID":"68dc8b8a-693f-436e-9219-f91abd6b1b94","Type":"ContainerDied","Data":"f71bac934edb82a80b63691bc6856052fd0620dadfcbca2b170467d961bf71dc"}
Oct 30 20:13:45.954243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:45.954212    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:46.040640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.040552    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-xks-ca\") pod \"3b81735f-d028-4a80-85d4-57cfb011c4b2\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") "
Oct 30 20:13:46.040812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.040690    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-openshift-ca\") pod \"3b81735f-d028-4a80-85d4-57cfb011c4b2\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") "
Oct 30 20:13:46.040812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.040713    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b81735f-d028-4a80-85d4-57cfb011c4b2-kube-api-access-bkbrf\" (UniqueName: \"kubernetes.io/projected/3b81735f-d028-4a80-85d4-57cfb011c4b2-kube-api-access-bkbrf\") pod \"3b81735f-d028-4a80-85d4-57cfb011c4b2\" (UID: \"3b81735f-d028-4a80-85d4-57cfb011c4b2\") "
Oct 30 20:13:46.041482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.041176    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-openshift-ca" pod "3b81735f-d028-4a80-85d4-57cfb011c4b2" (UID: "3b81735f-d028-4a80-85d4-57cfb011c4b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:46.041949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.041921    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:46.042907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.042878    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-xks-ca" pod "3b81735f-d028-4a80-85d4-57cfb011c4b2" (UID: "3b81735f-d028-4a80-85d4-57cfb011c4b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:46.043215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.043192    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b81735f-d028-4a80-85d4-57cfb011c4b2-kube-api-access-bkbrf" pod "3b81735f-d028-4a80-85d4-57cfb011c4b2" (UID: "3b81735f-d028-4a80-85d4-57cfb011c4b2"). InnerVolumeSpecName "kube-api-access-bkbrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:46.142998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.142963    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b81735f-d028-4a80-85d4-57cfb011c4b2-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:46.142998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.142993    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkbrf\" (UniqueName: \"kubernetes.io/projected/3b81735f-d028-4a80-85d4-57cfb011c4b2-kube-api-access-bkbrf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:46.768894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.768854    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"
Oct 30 20:13:46.769357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.768866    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88" event={"ID":"3b81735f-d028-4a80-85d4-57cfb011c4b2","Type":"ContainerDied","Data":"14669acd9a4dfca2bdd9a054853c7612d63c05e1d6d6afd45f5ed8d02843f0ad"}
Oct 30 20:13:46.769357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.769019    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14669acd9a4dfca2bdd9a054853c7612d63c05e1d6d6afd45f5ed8d02843f0ad"
Oct 30 20:13:46.955086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:46.955052    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:47.088360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.088267    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-openshift-ca\") pod \"68dc8b8a-693f-436e-9219-f91abd6b1b94\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") "
Oct 30 20:13:47.088563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.088413    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-xks-ca\") pod \"68dc8b8a-693f-436e-9219-f91abd6b1b94\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") "
Oct 30 20:13:47.088563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.088513    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68dc8b8a-693f-436e-9219-f91abd6b1b94-kube-api-access-zg5mw\" (UniqueName: \"kubernetes.io/projected/68dc8b8a-693f-436e-9219-f91abd6b1b94-kube-api-access-zg5mw\") pod \"68dc8b8a-693f-436e-9219-f91abd6b1b94\" (UID: \"68dc8b8a-693f-436e-9219-f91abd6b1b94\") "
Oct 30 20:13:47.088737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.088709    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-openshift-ca" pod "68dc8b8a-693f-436e-9219-f91abd6b1b94" (UID: "68dc8b8a-693f-436e-9219-f91abd6b1b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:47.089255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.089221    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:47.090721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.090692    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-xks-ca" pod "68dc8b8a-693f-436e-9219-f91abd6b1b94" (UID: "68dc8b8a-693f-436e-9219-f91abd6b1b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:47.091113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.091086    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dc8b8a-693f-436e-9219-f91abd6b1b94-kube-api-access-zg5mw" pod "68dc8b8a-693f-436e-9219-f91abd6b1b94" (UID: "68dc8b8a-693f-436e-9219-f91abd6b1b94"). InnerVolumeSpecName "kube-api-access-zg5mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:47.190887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.190855    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68dc8b8a-693f-436e-9219-f91abd6b1b94-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:47.190887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.190889    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5mw\" (UniqueName: \"kubernetes.io/projected/68dc8b8a-693f-436e-9219-f91abd6b1b94-kube-api-access-zg5mw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:47.259011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.258976    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"]
Oct 30 20:13:47.782153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.782117    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"
Oct 30 20:13:47.782153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.782139    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs" event={"ID":"68dc8b8a-693f-436e-9219-f91abd6b1b94","Type":"ContainerDied","Data":"87c4789ae9ec0390d59e3b03b23006168ca2b56fe38ce036a2c50e98f6e5686b"}
Oct 30 20:13:47.782649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.782207    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c4789ae9ec0390d59e3b03b23006168ca2b56fe38ce036a2c50e98f6e5686b"
Oct 30 20:13:47.822243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.822215    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"]
Oct 30 20:13:47.822834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.822820    4519 cpu_manager.go:414] "containerMap: removing container" podUID="68dc8b8a-693f-436e-9219-f91abd6b1b94" containerName="revoke-keys"
Oct 30 20:13:47.822891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.822835    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68dc8b8a-693f-436e-9219-f91abd6b1b94" containerName="revoke-keys"
Oct 30 20:13:47.822891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.822869    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3b81735f-d028-4a80-85d4-57cfb011c4b2" containerName="revoke-keys"
Oct 30 20:13:47.822891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.822875    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b81735f-d028-4a80-85d4-57cfb011c4b2" containerName="revoke-keys"
Oct 30 20:13:47.823006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.822988    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b81735f-d028-4a80-85d4-57cfb011c4b2" containerName="revoke-keys"
Oct 30 20:13:47.823006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.823004    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="68dc8b8a-693f-436e-9219-f91abd6b1b94" containerName="revoke-keys"
Oct 30 20:13:47.826785 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.826761    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:47.841376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.841337    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:47.841562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.841533    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:13:47.845593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.845565    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"]
Oct 30 20:13:47.907532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.907429    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hmjk\" (UniqueName: \"kubernetes.io/projected/84947a8f-f3cc-43b2-a5cc-612bfda89855-kube-api-access-9hmjk\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:47.907690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.907661    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:47.907785 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:47.907770    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.009640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.009576    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.009845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.009707    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.009907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.009879    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hmjk\" (UniqueName: \"kubernetes.io/projected/84947a8f-f3cc-43b2-a5cc-612bfda89855-kube-api-access-9hmjk\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.010830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.010568    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.012944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.012918    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.031380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.031350    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hmjk\" (UniqueName: \"kubernetes.io/projected/84947a8f-f3cc-43b2-a5cc-612bfda89855-kube-api-access-9hmjk\") pod \"maas-api-revoke-sub-8945b2f7-g7pj7\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.144575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.144540    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:48.456517 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:13:48.456447    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84947a8f_f3cc_43b2_a5cc_612bfda89855.slice/crio-e99ed41e59205ed11bc98a5cc293872976670881f027e92f0d802e12c5f01bc9 WatchSource:0}: Error finding container e99ed41e59205ed11bc98a5cc293872976670881f027e92f0d802e12c5f01bc9: Status 404 returned error can't find the container with id e99ed41e59205ed11bc98a5cc293872976670881f027e92f0d802e12c5f01bc9
Oct 30 20:13:48.456674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.456653    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"]
Oct 30 20:13:48.792419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:48.792333    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7" event={"ID":"84947a8f-f3cc-43b2-a5cc-612bfda89855","Type":"ContainerStarted","Data":"e99ed41e59205ed11bc98a5cc293872976670881f027e92f0d802e12c5f01bc9"}
Oct 30 20:13:49.801176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:49.801081    4519 generic.go:356] "Generic (PLEG): container finished" podID="84947a8f-f3cc-43b2-a5cc-612bfda89855" containerID="0531bfa8522d26b66f861d28fc8fe6965f699530555603de92dd6fc9f9267b78" exitCode=0
Oct 30 20:13:49.801176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:49.801150    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7" event={"ID":"84947a8f-f3cc-43b2-a5cc-612bfda89855","Type":"ContainerDied","Data":"0531bfa8522d26b66f861d28fc8fe6965f699530555603de92dd6fc9f9267b78"}
Oct 30 20:13:50.988998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:50.988974    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:51.054272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.054240    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-openshift-ca\") pod \"84947a8f-f3cc-43b2-a5cc-612bfda89855\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") "
Oct 30 20:13:51.054601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.054574    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84947a8f-f3cc-43b2-a5cc-612bfda89855-kube-api-access-9hmjk\" (UniqueName: \"kubernetes.io/projected/84947a8f-f3cc-43b2-a5cc-612bfda89855-kube-api-access-9hmjk\") pod \"84947a8f-f3cc-43b2-a5cc-612bfda89855\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") "
Oct 30 20:13:51.055016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.054996    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-xks-ca\") pod \"84947a8f-f3cc-43b2-a5cc-612bfda89855\" (UID: \"84947a8f-f3cc-43b2-a5cc-612bfda89855\") "
Oct 30 20:13:51.055400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.054641    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-openshift-ca" pod "84947a8f-f3cc-43b2-a5cc-612bfda89855" (UID: "84947a8f-f3cc-43b2-a5cc-612bfda89855"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:51.056028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.056004    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:51.058312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.058284    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84947a8f-f3cc-43b2-a5cc-612bfda89855-kube-api-access-9hmjk" pod "84947a8f-f3cc-43b2-a5cc-612bfda89855" (UID: "84947a8f-f3cc-43b2-a5cc-612bfda89855"). InnerVolumeSpecName "kube-api-access-9hmjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:51.058536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.058508    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-xks-ca" pod "84947a8f-f3cc-43b2-a5cc-612bfda89855" (UID: "84947a8f-f3cc-43b2-a5cc-612bfda89855"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:51.157811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.157775    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84947a8f-f3cc-43b2-a5cc-612bfda89855-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:51.157811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.157805    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hmjk\" (UniqueName: \"kubernetes.io/projected/84947a8f-f3cc-43b2-a5cc-612bfda89855-kube-api-access-9hmjk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:13:51.913560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.901442    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"
Oct 30 20:13:51.917676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.917619    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7" event={"ID":"84947a8f-f3cc-43b2-a5cc-612bfda89855","Type":"ContainerDied","Data":"e99ed41e59205ed11bc98a5cc293872976670881f027e92f0d802e12c5f01bc9"}
Oct 30 20:13:51.917872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:51.917857    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99ed41e59205ed11bc98a5cc293872976670881f027e92f0d802e12c5f01bc9"
Oct 30 20:13:52.759554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:52.759513    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"]
Oct 30 20:13:52.769384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:52.769355    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-936f70-ms8ct"]
Oct 30 20:13:53.613750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:53.613713    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19147efd-9c8f-4feb-981b-99dbedbe7d40" path="/var/lib/kubelet/pods/19147efd-9c8f-4feb-981b-99dbedbe7d40/volumes"
Oct 30 20:13:59.643363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:59.643314    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"]
Oct 30 20:13:59.643928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:13:59.643568    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" containerID="cri-o://6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d" gracePeriod=30
Oct 30 20:14:01.388612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.388576    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"]
Oct 30 20:14:01.389176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.389160    4519 cpu_manager.go:414] "containerMap: removing container" podUID="84947a8f-f3cc-43b2-a5cc-612bfda89855" containerName="revoke-keys"
Oct 30 20:14:01.389176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.389175    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84947a8f-f3cc-43b2-a5cc-612bfda89855" containerName="revoke-keys"
Oct 30 20:14:01.389304 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.389294    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="84947a8f-f3cc-43b2-a5cc-612bfda89855" containerName="revoke-keys"
Oct 30 20:14:01.394368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.394346    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.413444 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.413418    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"]
Oct 30 20:14:01.417821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.417796    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:14:01.417935 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.417912    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:14:01.487859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.487820    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.487859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.487863    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89c5j\" (UniqueName: \"kubernetes.io/projected/008571ad-6703-4b26-9ff7-6cdc8533932a-kube-api-access-89c5j\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.488167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.488134    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.589653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.589615    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.589868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.589660    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89c5j\" (UniqueName: \"kubernetes.io/projected/008571ad-6703-4b26-9ff7-6cdc8533932a-kube-api-access-89c5j\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.589868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.589852    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.590455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.590425    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.592424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.592400    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.609274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.609253    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89c5j\" (UniqueName: \"kubernetes.io/projected/008571ad-6703-4b26-9ff7-6cdc8533932a-kube-api-access-89c5j\") pod \"maas-api-revoke-sub-1583ef96-4758m\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:01.714853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:01.714825    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:02.052501 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:14:02.052278    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008571ad_6703_4b26_9ff7_6cdc8533932a.slice/crio-66d56ebcdaf2ec877764931f29693102af1d04899afa9e16aeb79fb932472f86 WatchSource:0}: Error finding container 66d56ebcdaf2ec877764931f29693102af1d04899afa9e16aeb79fb932472f86: Status 404 returned error can't find the container with id 66d56ebcdaf2ec877764931f29693102af1d04899afa9e16aeb79fb932472f86
Oct 30 20:14:02.055819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.054628    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"]
Oct 30 20:14:02.072899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.072862    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m" event={"ID":"008571ad-6703-4b26-9ff7-6cdc8533932a","Type":"ContainerStarted","Data":"66d56ebcdaf2ec877764931f29693102af1d04899afa9e16aeb79fb932472f86"}
Oct 30 20:14:02.238160 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:02.237390    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:14:02.313799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.313716    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7" podUID="9a651deb-f472-4c76-aa17-151115629af0" containerName="main" containerID="cri-o://405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1" gracePeriod=45
Oct 30 20:14:02.619221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.618670    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"]
Oct 30 20:14:02.708401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.708375    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:14:02.716459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.716429    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-model-cache\") pod \"9a651deb-f472-4c76-aa17-151115629af0\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") "
Oct 30 20:14:02.716666 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.716559    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-dshm\") pod \"9a651deb-f472-4c76-aa17-151115629af0\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") "
Oct 30 20:14:02.716666 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.716628    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-home\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-home\") pod \"9a651deb-f472-4c76-aa17-151115629af0\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") "
Oct 30 20:14:02.716797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.716726    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a651deb-f472-4c76-aa17-151115629af0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a651deb-f472-4c76-aa17-151115629af0-tls-certs\") pod \"9a651deb-f472-4c76-aa17-151115629af0\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") "
Oct 30 20:14:02.716797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.716766    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a651deb-f472-4c76-aa17-151115629af0-kube-api-access-624hc\" (UniqueName: \"kubernetes.io/projected/9a651deb-f472-4c76-aa17-151115629af0-kube-api-access-624hc\") pod \"9a651deb-f472-4c76-aa17-151115629af0\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") "
Oct 30 20:14:02.716908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.716814    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-tmp-dir\") pod \"9a651deb-f472-4c76-aa17-151115629af0\" (UID: \"9a651deb-f472-4c76-aa17-151115629af0\") "
Oct 30 20:14:02.718161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.718132    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-model-cache" pod "9a651deb-f472-4c76-aa17-151115629af0" (UID: "9a651deb-f472-4c76-aa17-151115629af0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:02.718488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.718448    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-home" pod "9a651deb-f472-4c76-aa17-151115629af0" (UID: "9a651deb-f472-4c76-aa17-151115629af0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:02.718716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.718695    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-tmp-dir" pod "9a651deb-f472-4c76-aa17-151115629af0" (UID: "9a651deb-f472-4c76-aa17-151115629af0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:02.720767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.720740    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a651deb-f472-4c76-aa17-151115629af0-kube-api-access-624hc" pod "9a651deb-f472-4c76-aa17-151115629af0" (UID: "9a651deb-f472-4c76-aa17-151115629af0"). InnerVolumeSpecName "kube-api-access-624hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:02.726772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.726747    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a651deb-f472-4c76-aa17-151115629af0-tls-certs" pod "9a651deb-f472-4c76-aa17-151115629af0" (UID: "9a651deb-f472-4c76-aa17-151115629af0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:02.743053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.743015    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-dshm" pod "9a651deb-f472-4c76-aa17-151115629af0" (UID: "9a651deb-f472-4c76-aa17-151115629af0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:02.819459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.819294    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:02.819459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.819333    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:02.819459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.819346    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:02.819459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.819358    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a651deb-f472-4c76-aa17-151115629af0-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:02.819459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.819371    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-624hc\" (UniqueName: \"kubernetes.io/projected/9a651deb-f472-4c76-aa17-151115629af0-kube-api-access-624hc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:02.819459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:02.819385    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a651deb-f472-4c76-aa17-151115629af0-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:03.117952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.115238    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m" event={"ID":"008571ad-6703-4b26-9ff7-6cdc8533932a","Type":"ContainerStarted","Data":"c89c0f50a149a47847e0d153663d0fa208cdc09bc623285a014ff7497bfe7e79"}
Oct 30 20:14:03.157433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.157381    4519 generic.go:356] "Generic (PLEG): container finished" podID="9a651deb-f472-4c76-aa17-151115629af0" containerID="405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1" exitCode=0
Oct 30 20:14:03.158181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.157691    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7" event={"ID":"9a651deb-f472-4c76-aa17-151115629af0","Type":"ContainerDied","Data":"405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1"}
Oct 30 20:14:03.158381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.158362    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7" event={"ID":"9a651deb-f472-4c76-aa17-151115629af0","Type":"ContainerDied","Data":"b91e88269d4f76ff891624ad92de0b37da21ead78b4f0d037a7ce55e927040e8"}
Oct 30 20:14:03.158544 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.158528    4519 scope.go:122] "RemoveContainer" containerID="405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1"
Oct 30 20:14:03.158822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.157791    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"
Oct 30 20:14:03.239393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.239360    4519 scope.go:122] "RemoveContainer" containerID="405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1"
Oct 30 20:14:03.240262 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:03.240216    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1\": container with ID starting with 405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1 not found: ID does not exist" containerID="405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1"
Oct 30 20:14:03.240394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.240257    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1"} err="failed to get container status \"405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1\": rpc error: code = NotFound desc = could not find container \"405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1\": container with ID starting with 405468162f1849b167afe91131978060f8566f71c7c20c3bd37b2d878d5b44b1 not found: ID does not exist"
Oct 30 20:14:03.251058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.249594    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m" podStartSLOduration=2.249458464 podStartE2EDuration="2.249458464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:03.159863862 +0000 UTC m=+2367.660567587" watchObservedRunningTime="2026-10-30 20:14:03.249458464 +0000 UTC m=+2367.750162190"
Oct 30 20:14:03.268497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.265430    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"]
Oct 30 20:14:03.268497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.266339    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9a651deb-f472-4c76-aa17-151115629af0" containerName="main"
Oct 30 20:14:03.268497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.266359    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a651deb-f472-4c76-aa17-151115629af0" containerName="main"
Oct 30 20:14:03.268497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.266601    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a651deb-f472-4c76-aa17-151115629af0" containerName="main"
Oct 30 20:14:03.272154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.272130    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.288954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.288929    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"]
Oct 30 20:14:03.317487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.316174    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"]
Oct 30 20:14:03.321883 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.321861    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7b5635/test-model-7b5635-kserve-599f6865bc-whzt7"]
Oct 30 20:14:03.343575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.343532    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbcxl\" (UniqueName: \"kubernetes.io/projected/55399c07-fbfe-474d-af29-e9c06ef80592-kube-api-access-fbcxl\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.343747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.343709    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.344086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.344062    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.369190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.369112    4519 patch_prober.go:29] interesting pod/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:14:03.369190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.369171    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:14:03.445743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.445706    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbcxl\" (UniqueName: \"kubernetes.io/projected/55399c07-fbfe-474d-af29-e9c06ef80592-kube-api-access-fbcxl\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.445902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.445838    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.446123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.446100    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.446673 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.446646    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.449853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.448696    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.464930 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.464901    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbcxl\" (UniqueName: \"kubernetes.io/projected/55399c07-fbfe-474d-af29-e9c06ef80592-kube-api-access-fbcxl\") pod \"maas-api-revoke-sub-a47f32db-ltmt5\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.597646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.597611    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:03.615181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.615150    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a651deb-f472-4c76-aa17-151115629af0" path="/var/lib/kubelet/pods/9a651deb-f472-4c76-aa17-151115629af0/volumes"
Oct 30 20:14:03.923446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:03.923417    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"]
Oct 30 20:14:03.923858 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:14:03.923772    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55399c07_fbfe_474d_af29_e9c06ef80592.slice/crio-a756367e97a718b7efaa14195fc85c19ef856a02f4ed91a56a21ee3703a43661 WatchSource:0}: Error finding container a756367e97a718b7efaa14195fc85c19ef856a02f4ed91a56a21ee3703a43661: Status 404 returned error can't find the container with id a756367e97a718b7efaa14195fc85c19ef856a02f4ed91a56a21ee3703a43661
Oct 30 20:14:04.168053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.168010    4519 generic.go:356] "Generic (PLEG): container finished" podID="008571ad-6703-4b26-9ff7-6cdc8533932a" containerID="c89c0f50a149a47847e0d153663d0fa208cdc09bc623285a014ff7497bfe7e79" exitCode=0
Oct 30 20:14:04.168250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.168098    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m" event={"ID":"008571ad-6703-4b26-9ff7-6cdc8533932a","Type":"ContainerDied","Data":"c89c0f50a149a47847e0d153663d0fa208cdc09bc623285a014ff7497bfe7e79"}
Oct 30 20:14:04.170964 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.170897    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5" event={"ID":"55399c07-fbfe-474d-af29-e9c06ef80592","Type":"ContainerStarted","Data":"a756367e97a718b7efaa14195fc85c19ef856a02f4ed91a56a21ee3703a43661"}
Oct 30 20:14:04.940410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.940381    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:14:04.987976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.987943    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istiod-ca-cert\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.988133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988058    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-data\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.988133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988122    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-certs\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.988240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988166    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-podinfo\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.988240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988222    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-token\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.988392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988366    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-data" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:04.988475 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988385    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istiod-ca-cert" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:04.988475 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988428    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-kube-api-access-hnrq2\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-kube-api-access-hnrq2\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.988606 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988518    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-envoy\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.988669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988602    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-credential-socket\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.989008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988982    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-socket\") pod \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\" (UID: \"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e\") "
Oct 30 20:14:04.989210 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.988887    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-credential-socket" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:04.989310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.989180    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-certs" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:04.989527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.989504    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-socket" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:04.990402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.990370    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:04.990551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.990537    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:04.990657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.990645    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:04.990748 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.990736    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:04.990837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.990825    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:04.991784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.991714    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-envoy" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:04.991784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.991720    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-kube-api-access-hnrq2" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "kube-api-access-hnrq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:04.992047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.992023    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-podinfo" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:04.992761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:04.992735    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-token" pod "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" (UID: "71f9b0a1-0b9e-4287-9210-89ff7edd7e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:05.092175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.092145    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnrq2\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-kube-api-access-hnrq2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:05.092175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.092174    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:05.092360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.092185    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:05.092360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.092194    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:05.183285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.183251    4519 generic.go:356] "Generic (PLEG): container finished" podID="55399c07-fbfe-474d-af29-e9c06ef80592" containerID="23e965ba54a93fd131dbdbf8d4504c2732c287cec4d2a8ada8ba8b577434a931" exitCode=0
Oct 30 20:14:05.183495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.183347    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5" event={"ID":"55399c07-fbfe-474d-af29-e9c06ef80592","Type":"ContainerDied","Data":"23e965ba54a93fd131dbdbf8d4504c2732c287cec4d2a8ada8ba8b577434a931"}
Oct 30 20:14:05.185593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.185566    4519 generic.go:356] "Generic (PLEG): container finished" podID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerID="6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d" exitCode=0
Oct 30 20:14:05.185699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.185627    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"
Oct 30 20:14:05.185699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.185648    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" event={"ID":"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e","Type":"ContainerDied","Data":"6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d"}
Oct 30 20:14:05.185699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.185698    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l" event={"ID":"71f9b0a1-0b9e-4287-9210-89ff7edd7e8e","Type":"ContainerDied","Data":"1ab7a6883bb773c28db3c0394651e4da210d2ce06f7ecb95aeaf5eac924e505b"}
Oct 30 20:14:05.185825 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.185719    4519 scope.go:122] "RemoveContainer" containerID="6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d"
Oct 30 20:14:05.199960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.199944    4519 scope.go:122] "RemoveContainer" containerID="a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3"
Oct 30 20:14:05.214286 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.214267    4519 scope.go:122] "RemoveContainer" containerID="6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d"
Oct 30 20:14:05.214703 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:05.214680    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d\": container with ID starting with 6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d not found: ID does not exist" containerID="6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d"
Oct 30 20:14:05.214802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.214715    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d"} err="failed to get container status \"6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d\": rpc error: code = NotFound desc = could not find container \"6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d\": container with ID starting with 6c73be2036ecc50ab4af72dbf7b043d58a43e16131be769478c3eb1ecb21696d not found: ID does not exist"
Oct 30 20:14:05.214802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.214742    4519 scope.go:122] "RemoveContainer" containerID="a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3"
Oct 30 20:14:05.215113 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:05.215052    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3\": container with ID starting with a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3 not found: ID does not exist" containerID="a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3"
Oct 30 20:14:05.215113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.215078    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3"} err="failed to get container status \"a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3\": rpc error: code = NotFound desc = could not find container \"a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3\": container with ID starting with a69eedd2392c5e272f80a27a0d83460fab81afe39c07f8cf7e291be5126662a3 not found: ID does not exist"
Oct 30 20:14:05.272736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.272651    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"]
Oct 30 20:14:05.272736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.272711    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-936f70-openshift-default-74f857b9dc-hv57l"]
Oct 30 20:14:05.333437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.333413    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:05.399526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.399489    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008571ad-6703-4b26-9ff7-6cdc8533932a-kube-api-access-89c5j\" (UniqueName: \"kubernetes.io/projected/008571ad-6703-4b26-9ff7-6cdc8533932a-kube-api-access-89c5j\") pod \"008571ad-6703-4b26-9ff7-6cdc8533932a\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") "
Oct 30 20:14:05.399731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.399677    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-xks-ca\") pod \"008571ad-6703-4b26-9ff7-6cdc8533932a\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") "
Oct 30 20:14:05.399804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.399793    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-openshift-ca\") pod \"008571ad-6703-4b26-9ff7-6cdc8533932a\" (UID: \"008571ad-6703-4b26-9ff7-6cdc8533932a\") "
Oct 30 20:14:05.400649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.400617    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-openshift-ca" pod "008571ad-6703-4b26-9ff7-6cdc8533932a" (UID: "008571ad-6703-4b26-9ff7-6cdc8533932a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:05.403542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.403505    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-xks-ca" pod "008571ad-6703-4b26-9ff7-6cdc8533932a" (UID: "008571ad-6703-4b26-9ff7-6cdc8533932a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:05.403542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.403521    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008571ad-6703-4b26-9ff7-6cdc8533932a-kube-api-access-89c5j" pod "008571ad-6703-4b26-9ff7-6cdc8533932a" (UID: "008571ad-6703-4b26-9ff7-6cdc8533932a"). InnerVolumeSpecName "kube-api-access-89c5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:05.501562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.501529    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:05.501562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.501558    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008571ad-6703-4b26-9ff7-6cdc8533932a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:05.501562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.501569    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89c5j\" (UniqueName: \"kubernetes.io/projected/008571ad-6703-4b26-9ff7-6cdc8533932a-kube-api-access-89c5j\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:05.614996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:05.614911    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" path="/var/lib/kubelet/pods/71f9b0a1-0b9e-4287-9210-89ff7edd7e8e/volumes"
Oct 30 20:14:06.199183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.199150    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"
Oct 30 20:14:06.199183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.199173    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m" event={"ID":"008571ad-6703-4b26-9ff7-6cdc8533932a","Type":"ContainerDied","Data":"66d56ebcdaf2ec877764931f29693102af1d04899afa9e16aeb79fb932472f86"}
Oct 30 20:14:06.199830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.199245    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66d56ebcdaf2ec877764931f29693102af1d04899afa9e16aeb79fb932472f86"
Oct 30 20:14:06.369825 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.369793    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:06.519203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.519125    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-xks-ca\") pod \"55399c07-fbfe-474d-af29-e9c06ef80592\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") "
Oct 30 20:14:06.519203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.519166    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-openshift-ca\") pod \"55399c07-fbfe-474d-af29-e9c06ef80592\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") "
Oct 30 20:14:06.519506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.519265    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55399c07-fbfe-474d-af29-e9c06ef80592-kube-api-access-fbcxl\" (UniqueName: \"kubernetes.io/projected/55399c07-fbfe-474d-af29-e9c06ef80592-kube-api-access-fbcxl\") pod \"55399c07-fbfe-474d-af29-e9c06ef80592\" (UID: \"55399c07-fbfe-474d-af29-e9c06ef80592\") "
Oct 30 20:14:06.519674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.519633    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-openshift-ca" pod "55399c07-fbfe-474d-af29-e9c06ef80592" (UID: "55399c07-fbfe-474d-af29-e9c06ef80592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:06.519860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.519842    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:06.521350 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.521326    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-xks-ca" pod "55399c07-fbfe-474d-af29-e9c06ef80592" (UID: "55399c07-fbfe-474d-af29-e9c06ef80592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:06.521540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.521519    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55399c07-fbfe-474d-af29-e9c06ef80592-kube-api-access-fbcxl" pod "55399c07-fbfe-474d-af29-e9c06ef80592" (UID: "55399c07-fbfe-474d-af29-e9c06ef80592"). InnerVolumeSpecName "kube-api-access-fbcxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:06.622031 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.621985    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55399c07-fbfe-474d-af29-e9c06ef80592-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:06.622031 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:06.622027    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbcxl\" (UniqueName: \"kubernetes.io/projected/55399c07-fbfe-474d-af29-e9c06ef80592-kube-api-access-fbcxl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:07.209538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:07.209509    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"
Oct 30 20:14:07.209927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:07.209536    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5" event={"ID":"55399c07-fbfe-474d-af29-e9c06ef80592","Type":"ContainerDied","Data":"a756367e97a718b7efaa14195fc85c19ef856a02f4ed91a56a21ee3703a43661"}
Oct 30 20:14:07.209927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:07.209584    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a756367e97a718b7efaa14195fc85c19ef856a02f4ed91a56a21ee3703a43661"
Oct 30 20:14:07.639341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:07.639251    4519 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod2a1bce2e-135b-44ff-9cf3-0dba343d26ef"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod2a1bce2e-135b-44ff-9cf3-0dba343d26ef] : Timed out while waiting for systemd to remove kubepods-burstable-pod2a1bce2e_135b_44ff_9cf3_0dba343d26ef.slice"
Oct 30 20:14:07.639341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:07.639293    4519 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="2a1bce2e-135b-44ff-9cf3-0dba343d26ef" err="failed to delete cgroup paths for [kubepods burstable pod2a1bce2e-135b-44ff-9cf3-0dba343d26ef] : unable to destroy cgroup paths for cgroup [kubepods burstable pod2a1bce2e-135b-44ff-9cf3-0dba343d26ef] : Timed out while waiting for systemd to remove kubepods-burstable-pod2a1bce2e_135b_44ff_9cf3_0dba343d26ef.slice"
Oct 30 20:14:16.461944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.460965    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"]
Oct 30 20:14:16.461944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.461907    4519 cpu_manager.go:414] "containerMap: removing container" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy"
Oct 30 20:14:16.461944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.461930    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy"
Oct 30 20:14:16.462648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.461982    4519 cpu_manager.go:414] "containerMap: removing container" podUID="008571ad-6703-4b26-9ff7-6cdc8533932a" containerName="revoke-keys"
Oct 30 20:14:16.462648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.461990    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008571ad-6703-4b26-9ff7-6cdc8533932a" containerName="revoke-keys"
Oct 30 20:14:16.462648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.462006    4519 cpu_manager.go:414] "containerMap: removing container" podUID="55399c07-fbfe-474d-af29-e9c06ef80592" containerName="revoke-keys"
Oct 30 20:14:16.462648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.462015    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55399c07-fbfe-474d-af29-e9c06ef80592" containerName="revoke-keys"
Oct 30 20:14:16.462648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.462174    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="008571ad-6703-4b26-9ff7-6cdc8533932a" containerName="revoke-keys"
Oct 30 20:14:16.462648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.462187    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="55399c07-fbfe-474d-af29-e9c06ef80592" containerName="revoke-keys"
Oct 30 20:14:16.462648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.462201    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy"
Oct 30 20:14:16.470361 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.470336    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.490040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.490004    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"]
Oct 30 20:14:16.492349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.492310    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:14:16.492349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.492326    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:14:16.548772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.548732    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6sn\" (UniqueName: \"kubernetes.io/projected/3dea5e3b-5bc0-4352-92de-0554ded085b2-kube-api-access-sc6sn\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.548959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.548829    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.548959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.548947    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.650724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.650679    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6sn\" (UniqueName: \"kubernetes.io/projected/3dea5e3b-5bc0-4352-92de-0554ded085b2-kube-api-access-sc6sn\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.650907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.650756    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.650907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.650840    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.651590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.651561    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.653369 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.653341    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.670604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.670569    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6sn\" (UniqueName: \"kubernetes.io/projected/3dea5e3b-5bc0-4352-92de-0554ded085b2-kube-api-access-sc6sn\") pod \"maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:16.795723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:16.795618    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:17.132501 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:14:17.127784    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dea5e3b_5bc0_4352_92de_0554ded085b2.slice/crio-e480bec173c41097329b1238ef94db0f6f87cc3233489c915e27794201aec050 WatchSource:0}: Error finding container e480bec173c41097329b1238ef94db0f6f87cc3233489c915e27794201aec050: Status 404 returned error can't find the container with id e480bec173c41097329b1238ef94db0f6f87cc3233489c915e27794201aec050
Oct 30 20:14:17.132501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:17.131757    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"]
Oct 30 20:14:17.316118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:17.316082    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs" event={"ID":"3dea5e3b-5bc0-4352-92de-0554ded085b2","Type":"ContainerStarted","Data":"e480bec173c41097329b1238ef94db0f6f87cc3233489c915e27794201aec050"}
Oct 30 20:14:17.705939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:17.705856    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh" podUID="a0f21867-bf7c-4226-b673-bae8ad29600b" containerName="main" containerID="cri-o://fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201" gracePeriod=45
Oct 30 20:14:17.986085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:17.986050    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:14:18.078989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.078901    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-tmp-dir\") pod \"a0f21867-bf7c-4226-b673-bae8ad29600b\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") "
Oct 30 20:14:18.079157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.079023    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-dshm\") pod \"a0f21867-bf7c-4226-b673-bae8ad29600b\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") "
Oct 30 20:14:18.079157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.079126    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-model-cache\") pod \"a0f21867-bf7c-4226-b673-bae8ad29600b\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") "
Oct 30 20:14:18.079278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.079202    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-home\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-home\") pod \"a0f21867-bf7c-4226-b673-bae8ad29600b\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") "
Oct 30 20:14:18.079278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.079230    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-tmp-dir" pod "a0f21867-bf7c-4226-b673-bae8ad29600b" (UID: "a0f21867-bf7c-4226-b673-bae8ad29600b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:18.079278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.079276    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f21867-bf7c-4226-b673-bae8ad29600b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f21867-bf7c-4226-b673-bae8ad29600b-tls-certs\") pod \"a0f21867-bf7c-4226-b673-bae8ad29600b\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") "
Oct 30 20:14:18.079452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.079296    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f21867-bf7c-4226-b673-bae8ad29600b-kube-api-access-gqjhg\" (UniqueName: \"kubernetes.io/projected/a0f21867-bf7c-4226-b673-bae8ad29600b-kube-api-access-gqjhg\") pod \"a0f21867-bf7c-4226-b673-bae8ad29600b\" (UID: \"a0f21867-bf7c-4226-b673-bae8ad29600b\") "
Oct 30 20:14:18.079877 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.079795    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-home" pod "a0f21867-bf7c-4226-b673-bae8ad29600b" (UID: "a0f21867-bf7c-4226-b673-bae8ad29600b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:18.080084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.080060    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-model-cache" pod "a0f21867-bf7c-4226-b673-bae8ad29600b" (UID: "a0f21867-bf7c-4226-b673-bae8ad29600b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:18.080152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.080123    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:18.080152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.080140    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:18.081978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.081953    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-dshm" pod "a0f21867-bf7c-4226-b673-bae8ad29600b" (UID: "a0f21867-bf7c-4226-b673-bae8ad29600b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:18.082662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.082636    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f21867-bf7c-4226-b673-bae8ad29600b-kube-api-access-gqjhg" pod "a0f21867-bf7c-4226-b673-bae8ad29600b" (UID: "a0f21867-bf7c-4226-b673-bae8ad29600b"). InnerVolumeSpecName "kube-api-access-gqjhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:18.082782 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.082763    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f21867-bf7c-4226-b673-bae8ad29600b-tls-certs" pod "a0f21867-bf7c-4226-b673-bae8ad29600b" (UID: "a0f21867-bf7c-4226-b673-bae8ad29600b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:18.181138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.181097    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:18.181138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.181127    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f21867-bf7c-4226-b673-bae8ad29600b-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:18.181138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.181140    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f21867-bf7c-4226-b673-bae8ad29600b-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:18.181138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.181149    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjhg\" (UniqueName: \"kubernetes.io/projected/a0f21867-bf7c-4226-b673-bae8ad29600b-kube-api-access-gqjhg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:18.326288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.326247    4519 generic.go:356] "Generic (PLEG): container finished" podID="3dea5e3b-5bc0-4352-92de-0554ded085b2" containerID="17feb76aefd77e129980cfae6620591190e3af9254866b0ec1510ba6aadf4f5e" exitCode=0
Oct 30 20:14:18.326502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.326303    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs" event={"ID":"3dea5e3b-5bc0-4352-92de-0554ded085b2","Type":"ContainerDied","Data":"17feb76aefd77e129980cfae6620591190e3af9254866b0ec1510ba6aadf4f5e"}
Oct 30 20:14:18.327864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.327842    4519 generic.go:356] "Generic (PLEG): container finished" podID="a0f21867-bf7c-4226-b673-bae8ad29600b" containerID="fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201" exitCode=0
Oct 30 20:14:18.327986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.327882    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"
Oct 30 20:14:18.327986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.327923    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh" event={"ID":"a0f21867-bf7c-4226-b673-bae8ad29600b","Type":"ContainerDied","Data":"fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201"}
Oct 30 20:14:18.327986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.327964    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh" event={"ID":"a0f21867-bf7c-4226-b673-bae8ad29600b","Type":"ContainerDied","Data":"46f4e7f4ca77eb4dbb2d35752fc4d715bcfa50a83d2bb47ec0392c2a14893771"}
Oct 30 20:14:18.327986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.327982    4519 scope.go:122] "RemoveContainer" containerID="fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201"
Oct 30 20:14:18.339492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.339476    4519 scope.go:122] "RemoveContainer" containerID="fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201"
Oct 30 20:14:18.339711 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:18.339691    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201\": container with ID starting with fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201 not found: ID does not exist" containerID="fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201"
Oct 30 20:14:18.339790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.339734    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201"} err="failed to get container status \"fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201\": rpc error: code = NotFound desc = could not find container \"fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201\": container with ID starting with fb64316316ae750da1f1abb5836c4b0c47e28521589390739de6f1dc5ec7d201 not found: ID does not exist"
Oct 30 20:14:18.405260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.405223    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"]
Oct 30 20:14:18.413455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:18.413422    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ebe57b/test-model-ebe57b-kserve-67d88b46dc-zvllh"]
Oct 30 20:14:19.175266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.175226    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"]
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.581513    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"]
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.582861    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a0f21867-bf7c-4226-b673-bae8ad29600b" containerName="main"
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.582885    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f21867-bf7c-4226-b673-bae8ad29600b" containerName="main"
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.582934    4519 cpu_manager.go:414] "containerMap: removing container" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy"
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.582942    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy"
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.583140    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f9b0a1-0b9e-4287-9210-89ff7edd7e8e" containerName="istio-proxy"
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.583157    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f21867-bf7c-4226-b673-bae8ad29600b" containerName="main"
Oct 30 20:14:19.600529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.598832    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.615045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.608567    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"]
Oct 30 20:14:19.639108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.639013    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f21867-bf7c-4226-b673-bae8ad29600b" path="/var/lib/kubelet/pods/a0f21867-bf7c-4226-b673-bae8ad29600b/volumes"
Oct 30 20:14:19.652478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.647236    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m9j\" (UniqueName: \"kubernetes.io/projected/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-kube-api-access-k7m9j\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.652478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.647307    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.652478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.647628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.724225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.724198    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:19.749235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.749208    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dea5e3b-5bc0-4352-92de-0554ded085b2-kube-api-access-sc6sn\" (UniqueName: \"kubernetes.io/projected/3dea5e3b-5bc0-4352-92de-0554ded085b2-kube-api-access-sc6sn\") pod \"3dea5e3b-5bc0-4352-92de-0554ded085b2\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") "
Oct 30 20:14:19.749655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.749632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-xks-ca\") pod \"3dea5e3b-5bc0-4352-92de-0554ded085b2\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") "
Oct 30 20:14:19.762107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.762080    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-openshift-ca\") pod \"3dea5e3b-5bc0-4352-92de-0554ded085b2\" (UID: \"3dea5e3b-5bc0-4352-92de-0554ded085b2\") "
Oct 30 20:14:19.781914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.781881    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.783144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.783120    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m9j\" (UniqueName: \"kubernetes.io/projected/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-kube-api-access-k7m9j\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.783673 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.782906    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.784285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.763602    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dea5e3b-5bc0-4352-92de-0554ded085b2-kube-api-access-sc6sn" pod "3dea5e3b-5bc0-4352-92de-0554ded085b2" (UID: "3dea5e3b-5bc0-4352-92de-0554ded085b2"). InnerVolumeSpecName "kube-api-access-sc6sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:19.784493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.764185    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-xks-ca" pod "3dea5e3b-5bc0-4352-92de-0554ded085b2" (UID: "3dea5e3b-5bc0-4352-92de-0554ded085b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:19.784614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.781276    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-openshift-ca" pod "3dea5e3b-5bc0-4352-92de-0554ded085b2" (UID: "3dea5e3b-5bc0-4352-92de-0554ded085b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:19.784728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.783821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.785055 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.785034    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc6sn\" (UniqueName: \"kubernetes.io/projected/3dea5e3b-5bc0-4352-92de-0554ded085b2-kube-api-access-sc6sn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:19.785174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.785160    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:19.785269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.785258    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5e3b-5bc0-4352-92de-0554ded085b2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:19.791380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.789083    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.808515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.806498    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m9j\" (UniqueName: \"kubernetes.io/projected/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-kube-api-access-k7m9j\") pod \"maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:19.946296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:19.946257    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:20.284986 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:14:20.284948    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae44e2a_c918_4f0d_b5e0_094e0995f68a.slice/crio-53a190af22879597d52b653090186d858f8c6ddde56f21d4d3385e8bb061e4e9 WatchSource:0}: Error finding container 53a190af22879597d52b653090186d858f8c6ddde56f21d4d3385e8bb061e4e9: Status 404 returned error can't find the container with id 53a190af22879597d52b653090186d858f8c6ddde56f21d4d3385e8bb061e4e9
Oct 30 20:14:20.286248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:20.286218    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"]
Oct 30 20:14:20.458946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:20.458906    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9" event={"ID":"2ae44e2a-c918-4f0d-b5e0-094e0995f68a","Type":"ContainerStarted","Data":"53a190af22879597d52b653090186d858f8c6ddde56f21d4d3385e8bb061e4e9"}
Oct 30 20:14:20.460667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:20.460639    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs" event={"ID":"3dea5e3b-5bc0-4352-92de-0554ded085b2","Type":"ContainerDied","Data":"e480bec173c41097329b1238ef94db0f6f87cc3233489c915e27794201aec050"}
Oct 30 20:14:20.460667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:20.460658    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"
Oct 30 20:14:20.460837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:20.460685    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e480bec173c41097329b1238ef94db0f6f87cc3233489c915e27794201aec050"
Oct 30 20:14:21.368303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.368258    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"]
Oct 30 20:14:21.369359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.369327    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3dea5e3b-5bc0-4352-92de-0554ded085b2" containerName="revoke-keys"
Oct 30 20:14:21.369359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.369356    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dea5e3b-5bc0-4352-92de-0554ded085b2" containerName="revoke-keys"
Oct 30 20:14:21.369615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.369575    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dea5e3b-5bc0-4352-92de-0554ded085b2" containerName="revoke-keys"
Oct 30 20:14:21.374508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.374420    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.408000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.407967    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"]
Oct 30 20:14:21.410655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.410622    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a165e5\"/\"default-dockercfg-brp2f\""
Oct 30 20:14:21.410655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.410650    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a165e5\"/\"openshift-service-ca.crt\""
Oct 30 20:14:21.410848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.410696    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a165e5\"/\"kube-root-ca.crt\""
Oct 30 20:14:21.410848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.410747    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a165e5\"/\"auth-test-model-a165e5-kserve-self-signed-certs\""
Oct 30 20:14:21.478533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.477997    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9" event={"ID":"2ae44e2a-c918-4f0d-b5e0-094e0995f68a","Type":"ContainerStarted","Data":"c56914748c6e51d94cb8c56154ef90c0dc80ff5e3868b425995bb0ef40b0c63e"}
Oct 30 20:14:21.527327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.527290    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-home\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.527541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.527343    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tmp-dir\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.527541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.527493    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-model-cache\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.527692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.527628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7vc\" (UniqueName: \"kubernetes.io/projected/5d841ab1-9f09-4836-894d-8fa3a3cb3803-kube-api-access-jx7vc\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.527758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.527718    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tls-certs\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.527852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.527833    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-dshm\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.531822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.531772    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9" podStartSLOduration=2.5317615780000002 podStartE2EDuration="2.531761578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:21.52937012 +0000 UTC m=+2386.030073845" watchObservedRunningTime="2026-10-30 20:14:21.531761578 +0000 UTC m=+2386.032465301"
Oct 30 20:14:21.634599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.630901    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-home\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.634599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.630985    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tmp-dir\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.634599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.631106    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-model-cache\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.634599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.631258    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7vc\" (UniqueName: \"kubernetes.io/projected/5d841ab1-9f09-4836-894d-8fa3a3cb3803-kube-api-access-jx7vc\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.634599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.631370    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tls-certs\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.634599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.631611    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-dshm\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.635120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.635044    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-home\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.644608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.635287    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tmp-dir\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.644608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.635551    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-model-cache\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.652180 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.647993    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-dshm\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.682488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.671848    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tls-certs\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.690235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.690163    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7vc\" (UniqueName: \"kubernetes.io/projected/5d841ab1-9f09-4836-894d-8fa3a3cb3803-kube-api-access-jx7vc\") pod \"auth-test-model-a165e5-kserve-58988f865-8x96l\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") " pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:21.718004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:21.717965    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:22.791756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:22.791537    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"]
Oct 30 20:14:22.802309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:22.802284    4519 generic.go:356] "Generic (PLEG): container finished" podID="2ae44e2a-c918-4f0d-b5e0-094e0995f68a" containerID="c56914748c6e51d94cb8c56154ef90c0dc80ff5e3868b425995bb0ef40b0c63e" exitCode=0
Oct 30 20:14:22.802545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:22.802530    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9" event={"ID":"2ae44e2a-c918-4f0d-b5e0-094e0995f68a","Type":"ContainerDied","Data":"c56914748c6e51d94cb8c56154ef90c0dc80ff5e3868b425995bb0ef40b0c63e"}
Oct 30 20:14:23.812689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:23.812648    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l" event={"ID":"5d841ab1-9f09-4836-894d-8fa3a3cb3803","Type":"ContainerStarted","Data":"a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062"}
Oct 30 20:14:23.812689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:23.812693    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l" event={"ID":"5d841ab1-9f09-4836-894d-8fa3a3cb3803","Type":"ContainerStarted","Data":"cb3ec2f4a6780096522e88cab1c886c19dd0e7dc6efaff0292c2ea42800634f7"}
Oct 30 20:14:23.862045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:23.861992    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l" podStartSLOduration=2.861977741 podStartE2EDuration="2.861977741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:23.858625829 +0000 UTC m=+2388.359329553" watchObservedRunningTime="2026-10-30 20:14:23.861977741 +0000 UTC m=+2388.362681490"
Oct 30 20:14:24.033197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.033162    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:24.126057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.125956    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-openshift-ca\") pod \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") "
Oct 30 20:14:24.126241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.126096    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-xks-ca\") pod \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") "
Oct 30 20:14:24.126324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.126277    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-kube-api-access-k7m9j\" (UniqueName: \"kubernetes.io/projected/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-kube-api-access-k7m9j\") pod \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\" (UID: \"2ae44e2a-c918-4f0d-b5e0-094e0995f68a\") "
Oct 30 20:14:24.127512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.127483    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-openshift-ca" pod "2ae44e2a-c918-4f0d-b5e0-094e0995f68a" (UID: "2ae44e2a-c918-4f0d-b5e0-094e0995f68a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:24.135264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.135240    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-xks-ca" pod "2ae44e2a-c918-4f0d-b5e0-094e0995f68a" (UID: "2ae44e2a-c918-4f0d-b5e0-094e0995f68a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:24.136009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.135980    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-kube-api-access-k7m9j" pod "2ae44e2a-c918-4f0d-b5e0-094e0995f68a" (UID: "2ae44e2a-c918-4f0d-b5e0-094e0995f68a"). InnerVolumeSpecName "kube-api-access-k7m9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:24.228967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.228933    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7m9j\" (UniqueName: \"kubernetes.io/projected/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-kube-api-access-k7m9j\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:24.228967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.228962    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:24.228967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.228973    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae44e2a-c918-4f0d-b5e0-094e0995f68a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:24.718731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.718697    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:24.822641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.822608    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"
Oct 30 20:14:24.822641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.822635    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9" event={"ID":"2ae44e2a-c918-4f0d-b5e0-094e0995f68a","Type":"ContainerDied","Data":"53a190af22879597d52b653090186d858f8c6ddde56f21d4d3385e8bb061e4e9"}
Oct 30 20:14:24.823117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:24.822689    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a190af22879597d52b653090186d858f8c6ddde56f21d4d3385e8bb061e4e9"
Oct 30 20:14:31.648495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:31.647565    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"]
Oct 30 20:14:31.718788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:31.718745    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:31.960661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:31.960625    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:31.986958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:31.986928    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:14:32.116289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.116253    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"]
Oct 30 20:14:32.120978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.120953    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2ae44e2a-c918-4f0d-b5e0-094e0995f68a" containerName="revoke-keys"
Oct 30 20:14:32.121191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.121161    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae44e2a-c918-4f0d-b5e0-094e0995f68a" containerName="revoke-keys"
Oct 30 20:14:32.121579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.121545    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae44e2a-c918-4f0d-b5e0-094e0995f68a" containerName="revoke-keys"
Oct 30 20:14:32.149485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.148344    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.174905 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.166016    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:14:32.174905 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.166092    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:14:32.246996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.242231    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"]
Oct 30 20:14:32.274489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.272435    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzqb\" (UniqueName: \"kubernetes.io/projected/d21ee149-885e-4f4f-b794-eba65e0040e5-kube-api-access-dgzqb\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.274489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.272541    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.274489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.272593    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.331099 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:32.331060    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:14:32.374163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.374111    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzqb\" (UniqueName: \"kubernetes.io/projected/d21ee149-885e-4f4f-b794-eba65e0040e5-kube-api-access-dgzqb\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.374356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.374220    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.374356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.374278    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.375861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.375832    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.386352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.386326    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.395324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.395296    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzqb\" (UniqueName: \"kubernetes.io/projected/d21ee149-885e-4f4f-b794-eba65e0040e5-kube-api-access-dgzqb\") pod \"maas-api-revoke-keys-e2e-inf-a-c17154-brnft\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.487155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.487115    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:32.820751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.820721    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"]
Oct 30 20:14:32.921154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:32.921123    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft" event={"ID":"d21ee149-885e-4f4f-b794-eba65e0040e5","Type":"ContainerStarted","Data":"dba7c78924a8cc867b1e0122292d3829a1e78aab90f33d44902888d3eb14bf16"}
Oct 30 20:14:33.935520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:33.935481    4519 generic.go:356] "Generic (PLEG): container finished" podID="d21ee149-885e-4f4f-b794-eba65e0040e5" containerID="0db15f5b0b09adfe3ef11605f008fc0daea63efe85bd09f47d1a382788909862" exitCode=0
Oct 30 20:14:33.935951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:33.935561    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft" event={"ID":"d21ee149-885e-4f4f-b794-eba65e0040e5","Type":"ContainerDied","Data":"0db15f5b0b09adfe3ef11605f008fc0daea63efe85bd09f47d1a382788909862"}
Oct 30 20:14:34.215718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.215676    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98" podUID="667ed976-af33-4e4b-b644-b7358acb616e" containerName="main" containerID="cri-o://07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339" gracePeriod=45
Oct 30 20:14:34.529107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.529081    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:14:34.614408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.614370    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-dshm\") pod \"667ed976-af33-4e4b-b644-b7358acb616e\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") "
Oct 30 20:14:34.614625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.614418    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-home\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-home\") pod \"667ed976-af33-4e4b-b644-b7358acb616e\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") "
Oct 30 20:14:34.614625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.614517    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-model-cache\") pod \"667ed976-af33-4e4b-b644-b7358acb616e\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") "
Oct 30 20:14:34.614779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.614644    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667ed976-af33-4e4b-b644-b7358acb616e-kube-api-access-s4jkr\" (UniqueName: \"kubernetes.io/projected/667ed976-af33-4e4b-b644-b7358acb616e-kube-api-access-s4jkr\") pod \"667ed976-af33-4e4b-b644-b7358acb616e\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") "
Oct 30 20:14:34.614779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.614734    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-tmp-dir\") pod \"667ed976-af33-4e4b-b644-b7358acb616e\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") "
Oct 30 20:14:34.614894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.614766    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-home" pod "667ed976-af33-4e4b-b644-b7358acb616e" (UID: "667ed976-af33-4e4b-b644-b7358acb616e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:34.614894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.614778    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/667ed976-af33-4e4b-b644-b7358acb616e-tls-certs\" (UniqueName: \"kubernetes.io/secret/667ed976-af33-4e4b-b644-b7358acb616e-tls-certs\") pod \"667ed976-af33-4e4b-b644-b7358acb616e\" (UID: \"667ed976-af33-4e4b-b644-b7358acb616e\") "
Oct 30 20:14:34.615160 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.615132    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-model-cache" pod "667ed976-af33-4e4b-b644-b7358acb616e" (UID: "667ed976-af33-4e4b-b644-b7358acb616e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:34.615307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.615286    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-tmp-dir" pod "667ed976-af33-4e4b-b644-b7358acb616e" (UID: "667ed976-af33-4e4b-b644-b7358acb616e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:34.615657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.615617    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:34.615657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.615638    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:34.615657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.615656    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:34.617439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.617412    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-dshm" pod "667ed976-af33-4e4b-b644-b7358acb616e" (UID: "667ed976-af33-4e4b-b644-b7358acb616e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:34.617704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.617678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667ed976-af33-4e4b-b644-b7358acb616e-kube-api-access-s4jkr" pod "667ed976-af33-4e4b-b644-b7358acb616e" (UID: "667ed976-af33-4e4b-b644-b7358acb616e"). InnerVolumeSpecName "kube-api-access-s4jkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:34.618914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.618878    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667ed976-af33-4e4b-b644-b7358acb616e-tls-certs" pod "667ed976-af33-4e4b-b644-b7358acb616e" (UID: "667ed976-af33-4e4b-b644-b7358acb616e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:34.717328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.717290    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667ed976-af33-4e4b-b644-b7358acb616e-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:34.717328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.717320    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667ed976-af33-4e4b-b644-b7358acb616e-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:34.717328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.717330    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4jkr\" (UniqueName: \"kubernetes.io/projected/667ed976-af33-4e4b-b644-b7358acb616e-kube-api-access-s4jkr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:34.954611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.954561    4519 generic.go:356] "Generic (PLEG): container finished" podID="667ed976-af33-4e4b-b644-b7358acb616e" containerID="07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339" exitCode=0
Oct 30 20:14:34.955087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.954642    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"
Oct 30 20:14:34.955087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.954662    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98" event={"ID":"667ed976-af33-4e4b-b644-b7358acb616e","Type":"ContainerDied","Data":"07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339"}
Oct 30 20:14:34.955087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.954718    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98" event={"ID":"667ed976-af33-4e4b-b644-b7358acb616e","Type":"ContainerDied","Data":"93db12053bffbd6465b8252532e5549df3f3b5cdefeb3df0d552fb91edb51776"}
Oct 30 20:14:34.955087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.954750    4519 scope.go:122] "RemoveContainer" containerID="07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339"
Oct 30 20:14:34.970651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.970618    4519 scope.go:122] "RemoveContainer" containerID="07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339"
Oct 30 20:14:34.971056 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:34.971029    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339\": container with ID starting with 07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339 not found: ID does not exist" containerID="07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339"
Oct 30 20:14:34.971141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:34.971065    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339"} err="failed to get container status \"07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339\": rpc error: code = NotFound desc = could not find container \"07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339\": container with ID starting with 07a60402560133afa5bdc302a92e44beef7bffe5c76bae3523023f39ad3bb339 not found: ID does not exist"
Oct 30 20:14:35.009158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.009127    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"]
Oct 30 20:14:35.021838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.021809    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98"]
Oct 30 20:14:35.136074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.136041    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:35.233866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.233786    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-openshift-ca\") pod \"d21ee149-885e-4f4f-b794-eba65e0040e5\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") "
Oct 30 20:14:35.234018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.233900    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21ee149-885e-4f4f-b794-eba65e0040e5-kube-api-access-dgzqb\" (UniqueName: \"kubernetes.io/projected/d21ee149-885e-4f4f-b794-eba65e0040e5-kube-api-access-dgzqb\") pod \"d21ee149-885e-4f4f-b794-eba65e0040e5\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") "
Oct 30 20:14:35.234018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.233925    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-xks-ca\") pod \"d21ee149-885e-4f4f-b794-eba65e0040e5\" (UID: \"d21ee149-885e-4f4f-b794-eba65e0040e5\") "
Oct 30 20:14:35.234240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.234212    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-openshift-ca" pod "d21ee149-885e-4f4f-b794-eba65e0040e5" (UID: "d21ee149-885e-4f4f-b794-eba65e0040e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:35.234532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.234515    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:35.236070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.236047    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-xks-ca" pod "d21ee149-885e-4f4f-b794-eba65e0040e5" (UID: "d21ee149-885e-4f4f-b794-eba65e0040e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:35.236141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.236064    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21ee149-885e-4f4f-b794-eba65e0040e5-kube-api-access-dgzqb" pod "d21ee149-885e-4f4f-b794-eba65e0040e5" (UID: "d21ee149-885e-4f4f-b794-eba65e0040e5"). InnerVolumeSpecName "kube-api-access-dgzqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:35.335408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.335371    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzqb\" (UniqueName: \"kubernetes.io/projected/d21ee149-885e-4f4f-b794-eba65e0040e5-kube-api-access-dgzqb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:35.335408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.335401    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ee149-885e-4f4f-b794-eba65e0040e5-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:35.508040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.508001    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-58d72c/test-model-58d72c-kserve-b489cdc57-mzh98" podUID="667ed976-af33-4e4b-b644-b7358acb616e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.207:8000/ready\": context deadline exceeded"
Oct 30 20:14:35.614561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.614524    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667ed976-af33-4e4b-b644-b7358acb616e" path="/var/lib/kubelet/pods/667ed976-af33-4e4b-b644-b7358acb616e/volumes"
Oct 30 20:14:35.968426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.968376    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft" event={"ID":"d21ee149-885e-4f4f-b794-eba65e0040e5","Type":"ContainerDied","Data":"dba7c78924a8cc867b1e0122292d3829a1e78aab90f33d44902888d3eb14bf16"}
Oct 30 20:14:35.969104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.968443    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dba7c78924a8cc867b1e0122292d3829a1e78aab90f33d44902888d3eb14bf16"
Oct 30 20:14:35.969104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:35.968542    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"
Oct 30 20:14:43.917192 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:43.917155    4519 scope.go:122] "RemoveContainer" containerID="272a3560738d300825719b656ddf5421accf3c4ff63fd74c1f061e633f705728"
Oct 30 20:14:46.697388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:46.697342    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9" podUID="6337d35e-be84-44cd-8b2e-556ff645482e" containerName="main" containerID="cri-o://e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611" gracePeriod=45
Oct 30 20:14:47.012361 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.012333    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:14:47.057739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.057708    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-dshm\") pod \"6337d35e-be84-44cd-8b2e-556ff645482e\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") "
Oct 30 20:14:47.057920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.057769    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-home\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-home\") pod \"6337d35e-be84-44cd-8b2e-556ff645482e\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") "
Oct 30 20:14:47.057920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.057850    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-tmp-dir\") pod \"6337d35e-be84-44cd-8b2e-556ff645482e\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") "
Oct 30 20:14:47.057920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.057891    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6337d35e-be84-44cd-8b2e-556ff645482e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6337d35e-be84-44cd-8b2e-556ff645482e-tls-certs\") pod \"6337d35e-be84-44cd-8b2e-556ff645482e\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") "
Oct 30 20:14:47.058100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.057920    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6337d35e-be84-44cd-8b2e-556ff645482e-kube-api-access-j7lkb\" (UniqueName: \"kubernetes.io/projected/6337d35e-be84-44cd-8b2e-556ff645482e-kube-api-access-j7lkb\") pod \"6337d35e-be84-44cd-8b2e-556ff645482e\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") "
Oct 30 20:14:47.058100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.058060    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-model-cache\") pod \"6337d35e-be84-44cd-8b2e-556ff645482e\" (UID: \"6337d35e-be84-44cd-8b2e-556ff645482e\") "
Oct 30 20:14:47.059271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.059241    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-model-cache" pod "6337d35e-be84-44cd-8b2e-556ff645482e" (UID: "6337d35e-be84-44cd-8b2e-556ff645482e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:47.061523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.061497    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-dshm" pod "6337d35e-be84-44cd-8b2e-556ff645482e" (UID: "6337d35e-be84-44cd-8b2e-556ff645482e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:47.061853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.061828    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-home" pod "6337d35e-be84-44cd-8b2e-556ff645482e" (UID: "6337d35e-be84-44cd-8b2e-556ff645482e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:47.062030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.062011    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-tmp-dir" pod "6337d35e-be84-44cd-8b2e-556ff645482e" (UID: "6337d35e-be84-44cd-8b2e-556ff645482e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:47.064742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.064701    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6337d35e-be84-44cd-8b2e-556ff645482e-kube-api-access-j7lkb" pod "6337d35e-be84-44cd-8b2e-556ff645482e" (UID: "6337d35e-be84-44cd-8b2e-556ff645482e"). InnerVolumeSpecName "kube-api-access-j7lkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:47.066219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.065620    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6337d35e-be84-44cd-8b2e-556ff645482e-tls-certs" pod "6337d35e-be84-44cd-8b2e-556ff645482e" (UID: "6337d35e-be84-44cd-8b2e-556ff645482e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:47.134250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.134208    4519 generic.go:356] "Generic (PLEG): container finished" podID="6337d35e-be84-44cd-8b2e-556ff645482e" containerID="e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611" exitCode=0
Oct 30 20:14:47.134430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.134275    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"
Oct 30 20:14:47.134430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.134284    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9" event={"ID":"6337d35e-be84-44cd-8b2e-556ff645482e","Type":"ContainerDied","Data":"e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611"}
Oct 30 20:14:47.134430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.134338    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9" event={"ID":"6337d35e-be84-44cd-8b2e-556ff645482e","Type":"ContainerDied","Data":"b636a2253188cfe98e62e798860969ef001600bb153545b986003b4cbb2748cf"}
Oct 30 20:14:47.134430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.134359    4519 scope.go:122] "RemoveContainer" containerID="e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611"
Oct 30 20:14:47.150029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.150008    4519 scope.go:122] "RemoveContainer" containerID="e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611"
Oct 30 20:14:47.150432 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:14:47.150402    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611\": container with ID starting with e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611 not found: ID does not exist" containerID="e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611"
Oct 30 20:14:47.150692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.150442    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611"} err="failed to get container status \"e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611\": rpc error: code = NotFound desc = could not find container \"e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611\": container with ID starting with e939497fdcac6fb5a5579001d703d6dc600d5b46be524dda3453b7a83132d611 not found: ID does not exist"
Oct 30 20:14:47.160253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.160230    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:47.160253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.160253    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:47.160408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.160289    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:47.160408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.160302    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337d35e-be84-44cd-8b2e-556ff645482e-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:47.160408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.160316    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7lkb\" (UniqueName: \"kubernetes.io/projected/6337d35e-be84-44cd-8b2e-556ff645482e-kube-api-access-j7lkb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:47.160408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.160330    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337d35e-be84-44cd-8b2e-556ff645482e-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:14:47.211587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.211556    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"]
Oct 30 20:14:47.224446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.224418    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6cd2ad/test-model-6cd2ad-kserve-7fbcdd6875-sfhf9"]
Oct 30 20:14:47.614188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:47.614155    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6337d35e-be84-44cd-8b2e-556ff645482e" path="/var/lib/kubelet/pods/6337d35e-be84-44cd-8b2e-556ff645482e/volumes"
Oct 30 20:14:50.459726 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:50.459687    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"]
Oct 30 20:14:50.464828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:50.464799    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b5635-hz4b9"]
Oct 30 20:14:51.613868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:51.613833    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae44e2a-c918-4f0d-b5e0-094e0995f68a" path="/var/lib/kubelet/pods/2ae44e2a-c918-4f0d-b5e0-094e0995f68a/volumes"
Oct 30 20:14:57.722144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:57.722107    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"]
Oct 30 20:14:57.722778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:57.722316    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" containerID="cri-o://e1dd10253e2cf36c600464a6ddccada56fcab741c4f4b75510866dd0c7088933" gracePeriod=30
Oct 30 20:14:58.346100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346064    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"]
Oct 30 20:14:58.346735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346711    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d21ee149-885e-4f4f-b794-eba65e0040e5" containerName="revoke-keys"
Oct 30 20:14:58.346735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346729    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21ee149-885e-4f4f-b794-eba65e0040e5" containerName="revoke-keys"
Oct 30 20:14:58.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346754    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6337d35e-be84-44cd-8b2e-556ff645482e" containerName="main"
Oct 30 20:14:58.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346760    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6337d35e-be84-44cd-8b2e-556ff645482e" containerName="main"
Oct 30 20:14:58.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346767    4519 cpu_manager.go:414] "containerMap: removing container" podUID="667ed976-af33-4e4b-b644-b7358acb616e" containerName="main"
Oct 30 20:14:58.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346772    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667ed976-af33-4e4b-b644-b7358acb616e" containerName="main"
Oct 30 20:14:58.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346908    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21ee149-885e-4f4f-b794-eba65e0040e5" containerName="revoke-keys"
Oct 30 20:14:58.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346924    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6337d35e-be84-44cd-8b2e-556ff645482e" containerName="main"
Oct 30 20:14:58.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.346931    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="667ed976-af33-4e4b-b644-b7358acb616e" containerName="main"
Oct 30 20:14:58.351983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.351964    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.370794 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.370769    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"]
Oct 30 20:14:58.385912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.385887    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:14:58.393775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.393752    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:14:58.529941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.529902    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtdh\" (UniqueName: \"kubernetes.io/projected/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-kube-api-access-tjtdh\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.530138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.530075    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.530138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.530113    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.632007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.631923    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtdh\" (UniqueName: \"kubernetes.io/projected/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-kube-api-access-tjtdh\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.632163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.632049    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.632163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.632086    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.632752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.632722    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.634706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.634678    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.652739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.652715    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtdh\" (UniqueName: \"kubernetes.io/projected/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-kube-api-access-tjtdh\") pod \"maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.667371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.667347    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:14:58.978806 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:14:58.978772    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c21fe7_0386_4554_bb49_63c3a43c9d8e.slice/crio-6cdcc145fc77c9dcda8cde649cb483caef8401922e84d31b1dced4f4f8b84d3e WatchSource:0}: Error finding container 6cdcc145fc77c9dcda8cde649cb483caef8401922e84d31b1dced4f4f8b84d3e: Status 404 returned error can't find the container with id 6cdcc145fc77c9dcda8cde649cb483caef8401922e84d31b1dced4f4f8b84d3e
Oct 30 20:14:58.981600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:58.981486    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"]
Oct 30 20:14:59.333196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:59.333153    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8" event={"ID":"a1c21fe7-0386-4554-bb49-63c3a43c9d8e","Type":"ContainerStarted","Data":"c43978cdf40727894416c6dc1a552bf20648209e92bad5e2fdfa6fc161cd4dc5"}
Oct 30 20:14:59.333388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:59.333225    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8" event={"ID":"a1c21fe7-0386-4554-bb49-63c3a43c9d8e","Type":"ContainerStarted","Data":"6cdcc145fc77c9dcda8cde649cb483caef8401922e84d31b1dced4f4f8b84d3e"}
Oct 30 20:14:59.395522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:14:59.392143    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8" podStartSLOduration=1.392123794 podStartE2EDuration="1.392123794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:59.386742663 +0000 UTC m=+2423.887446388" watchObservedRunningTime="2026-10-30 20:14:59.392123794 +0000 UTC m=+2423.892827520"
Oct 30 20:15:00.172672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.162081    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"]
Oct 30 20:15:00.172672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.172543    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.185264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.185231    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"]
Oct 30 20:15:00.203129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.203093    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.242224 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.242014    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"]
Oct 30 20:15:00.286506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.286163    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmpb\" (UniqueName: \"kubernetes.io/projected/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-kube-api-access-kwmpb\") pod \"maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.286506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.286270    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f3a318-c311-494c-8152-310d7c3b35e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.286506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.286422    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.286506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.286484    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd972\" (UniqueName: \"kubernetes.io/projected/a0f3a318-c311-494c-8152-310d7c3b35e6-kube-api-access-dd972\") pod \"maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.289275 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.289249    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.293554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.293518    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"]
Oct 30 20:15:00.300558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.300536    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.304611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.304548    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"]
Oct 30 20:15:00.310459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.310430    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.313590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.313561    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"]
Oct 30 20:15:00.346442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.346104    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.346827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.346020    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"]
Oct 30 20:15:00.352717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.352691    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"]
Oct 30 20:15:00.353726 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.353702    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.357633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.357611    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"]
Oct 30 20:15:00.366859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.366836    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"]
Oct 30 20:15:00.367903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.359115    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.370716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.364963    4519 generic.go:356] "Generic (PLEG): container finished" podID="a1c21fe7-0386-4554-bb49-63c3a43c9d8e" containerID="c43978cdf40727894416c6dc1a552bf20648209e92bad5e2fdfa6fc161cd4dc5" exitCode=0
Oct 30 20:15:00.374833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.372594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8" event={"ID":"a1c21fe7-0386-4554-bb49-63c3a43c9d8e","Type":"ContainerDied","Data":"c43978cdf40727894416c6dc1a552bf20648209e92bad5e2fdfa6fc161cd4dc5"}
Oct 30 20:15:00.374833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.372636    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"]
Oct 30 20:15:00.374833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.373863    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.377680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.377657    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"]
Oct 30 20:15:00.377772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.377759    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.382847 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.382586    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"]
Oct 30 20:15:00.387822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.387794    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmlw\" (UniqueName: \"kubernetes.io/projected/fa2a5dc7-e640-4fae-abec-0fe6f439634b-kube-api-access-xrmlw\") pod \"maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.388024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.387850    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"]
Oct 30 20:15:00.388024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388008    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.388215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.387921    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.388215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388096    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.388215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388167    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd972\" (UniqueName: \"kubernetes.io/projected/a0f3a318-c311-494c-8152-310d7c3b35e6-kube-api-access-dd972\") pod \"maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.388401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388296    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r87cm\" (UniqueName: \"kubernetes.io/projected/5a004868-5cba-4a74-83ca-2ef0975cee10-kube-api-access-r87cm\") pod \"maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.388401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388353    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a5dc7-e640-4fae-abec-0fe6f439634b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.388563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388441    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmpb\" (UniqueName: \"kubernetes.io/projected/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-kube-api-access-kwmpb\") pod \"maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.388648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388598    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f3a318-c311-494c-8152-310d7c3b35e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.388648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388635    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txsg\" (UniqueName: \"kubernetes.io/projected/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-kube-api-access-8txsg\") pod \"maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.388771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.388682    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a004868-5cba-4a74-83ca-2ef0975cee10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.389605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.389577    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.391184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.390763    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f3a318-c311-494c-8152-310d7c3b35e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.397143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.392446    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"]
Oct 30 20:15:00.398581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.397544    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"]
Oct 30 20:15:00.401114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.401097    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"]
Oct 30 20:15:00.408216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.408198    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"]
Oct 30 20:15:00.408430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.408413    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.409671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.409650    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"]
Oct 30 20:15:00.414906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.414311    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"]
Oct 30 20:15:00.419412 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.419392    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"]
Oct 30 20:15:00.424723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.424698    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"]
Oct 30 20:15:00.424897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.424798    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.425236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.425219    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmpb\" (UniqueName: \"kubernetes.io/projected/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-kube-api-access-kwmpb\") pod \"maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.425897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.425877    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd972\" (UniqueName: \"kubernetes.io/projected/a0f3a318-c311-494c-8152-310d7c3b35e6-kube-api-access-dd972\") pod \"maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.429191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.429173    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"]
Oct 30 20:15:00.434290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.434271    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"]
Oct 30 20:15:00.434382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.434366    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:00.437702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.437684    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"]
Oct 30 20:15:00.441816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.441795    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"]
Oct 30 20:15:00.445550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.445529    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"]
Oct 30 20:15:00.454119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.454094    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"]
Oct 30 20:15:00.459368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.459349    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"]
Oct 30 20:15:00.459496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.459454    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:00.465774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.465754    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"]
Oct 30 20:15:00.492297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492257    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14be7e43-88ac-4727-9c8d-cee1e4497ec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.492534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492511    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759hn\" (UniqueName: \"kubernetes.io/projected/a4a58aaa-0c5f-4aac-af12-43e70146549a-kube-api-access-759hn\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.492627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492555    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wlv\" (UniqueName: \"kubernetes.io/projected/944c5b58-189c-401a-aeea-2d1f58baeb46-kube-api-access-d8wlv\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.492627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492591    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6sr\" (UniqueName: \"kubernetes.io/projected/50cdcd80-a577-4bdb-8534-a60196f790e9-kube-api-access-kr6sr\") pod \"maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.492741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492645    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944c5b58-189c-401a-aeea-2d1f58baeb46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.492801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492718    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a58aaa-0c5f-4aac-af12-43e70146549a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.492801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492788    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txsg\" (UniqueName: \"kubernetes.io/projected/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-kube-api-access-8txsg\") pod \"maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.492921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492837    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a004868-5cba-4a74-83ca-2ef0975cee10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.492921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.492871    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmlw\" (UniqueName: \"kubernetes.io/projected/fa2a5dc7-e640-4fae-abec-0fe6f439634b-kube-api-access-xrmlw\") pod \"maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.493112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493085    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.493280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493260    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjk5c\" (UniqueName: \"kubernetes.io/projected/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-kube-api-access-zjk5c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.493364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493291    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzrj\" (UniqueName: \"kubernetes.io/projected/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-kube-api-access-kpzrj\") pod \"maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.493364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493316    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cdcd80-a577-4bdb-8534-a60196f790e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.493498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493366    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.493669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493646    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.493746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493701    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwwz\" (UniqueName: \"kubernetes.io/projected/14be7e43-88ac-4727-9c8d-cee1e4497ec4-kube-api-access-2wwwz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.493811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493758    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d56c5-518a-4275-b51f-4dd153ee64d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.493871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493824    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r87cm\" (UniqueName: \"kubernetes.io/projected/5a004868-5cba-4a74-83ca-2ef0975cee10-kube-api-access-r87cm\") pod \"maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.493931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493884    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a004868-5cba-4a74-83ca-2ef0975cee10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.493931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493890    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a5dc7-e640-4fae-abec-0fe6f439634b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.494051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493943    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.494051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.493977    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.494159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.494054    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rnq\" (UniqueName: \"kubernetes.io/projected/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-kube-api-access-f8rnq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.494224 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.494169    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfvn\" (UniqueName: \"kubernetes.io/projected/b65d56c5-518a-4275-b51f-4dd153ee64d3-kube-api-access-vzfvn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.494640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.494616    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a5dc7-e640-4fae-abec-0fe6f439634b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.523212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.523181    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r87cm\" (UniqueName: \"kubernetes.io/projected/5a004868-5cba-4a74-83ca-2ef0975cee10-kube-api-access-r87cm\") pod \"maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.524558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.524536    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmlw\" (UniqueName: \"kubernetes.io/projected/fa2a5dc7-e640-4fae-abec-0fe6f439634b-kube-api-access-xrmlw\") pod \"maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.524732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.524716    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txsg\" (UniqueName: \"kubernetes.io/projected/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-kube-api-access-8txsg\") pod \"maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.595845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.595754    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62cb1f4-31fd-4e48-9353-ec664afa53bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:00.595845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.595815    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.596090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.595886    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwwz\" (UniqueName: \"kubernetes.io/projected/14be7e43-88ac-4727-9c8d-cee1e4497ec4-kube-api-access-2wwwz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.596090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.595929    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d56c5-518a-4275-b51f-4dd153ee64d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.596090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.595986    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.596255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596146    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rnq\" (UniqueName: \"kubernetes.io/projected/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-kube-api-access-f8rnq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.596255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596188    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfvn\" (UniqueName: \"kubernetes.io/projected/b65d56c5-518a-4275-b51f-4dd153ee64d3-kube-api-access-vzfvn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.596255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596244    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14be7e43-88ac-4727-9c8d-cee1e4497ec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.596400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596286    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759hn\" (UniqueName: \"kubernetes.io/projected/a4a58aaa-0c5f-4aac-af12-43e70146549a-kube-api-access-759hn\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.596400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596320    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vzc\" (UniqueName: \"kubernetes.io/projected/dd6af51b-6caa-4f15-a151-dfc7928ddec8-kube-api-access-q5vzc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:00.596400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596352    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wlv\" (UniqueName: \"kubernetes.io/projected/944c5b58-189c-401a-aeea-2d1f58baeb46-kube-api-access-d8wlv\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.596400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596385    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6sr\" (UniqueName: \"kubernetes.io/projected/50cdcd80-a577-4bdb-8534-a60196f790e9-kube-api-access-kr6sr\") pod \"maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.596641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596411    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944c5b58-189c-401a-aeea-2d1f58baeb46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.596641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596435    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a58aaa-0c5f-4aac-af12-43e70146549a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.596641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596461    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6af51b-6caa-4f15-a151-dfc7928ddec8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:00.596641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596510    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4cv\" (UniqueName: \"kubernetes.io/projected/e62cb1f4-31fd-4e48-9353-ec664afa53bd-kube-api-access-wj4cv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:00.596845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596797    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.596904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596827    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d56c5-518a-4275-b51f-4dd153ee64d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.596904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596858    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.596904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596862    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.597071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596900    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjk5c\" (UniqueName: \"kubernetes.io/projected/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-kube-api-access-zjk5c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.597071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596938    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzrj\" (UniqueName: \"kubernetes.io/projected/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-kube-api-access-kpzrj\") pod \"maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.597071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.596966    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cdcd80-a577-4bdb-8534-a60196f790e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.597241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.597195    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14be7e43-88ac-4727-9c8d-cee1e4497ec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.597303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.597273    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944c5b58-189c-401a-aeea-2d1f58baeb46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.597514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.597495    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a58aaa-0c5f-4aac-af12-43e70146549a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.597599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.597545    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.597681 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.597664    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cdcd80-a577-4bdb-8534-a60196f790e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.598498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.598476    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:00.617017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.616984    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"]
Oct 30 20:15:00.617340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.617263    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:00.622026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.621677    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfvn\" (UniqueName: \"kubernetes.io/projected/b65d56c5-518a-4275-b51f-4dd153ee64d3-kube-api-access-vzfvn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.627619 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.627594    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6sr\" (UniqueName: \"kubernetes.io/projected/50cdcd80-a577-4bdb-8534-a60196f790e9-kube-api-access-kr6sr\") pod \"maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.627727 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.627617    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rnq\" (UniqueName: \"kubernetes.io/projected/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-kube-api-access-f8rnq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.627776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.627753    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzrj\" (UniqueName: \"kubernetes.io/projected/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-kube-api-access-kpzrj\") pod \"maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.628298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.628272    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwwz\" (UniqueName: \"kubernetes.io/projected/14be7e43-88ac-4727-9c8d-cee1e4497ec4-kube-api-access-2wwwz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.628397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.628348    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjk5c\" (UniqueName: \"kubernetes.io/projected/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-kube-api-access-zjk5c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.628825 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.628646    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-759hn\" (UniqueName: \"kubernetes.io/projected/a4a58aaa-0c5f-4aac-af12-43e70146549a-kube-api-access-759hn\") pod \"maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.628948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.628928    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wlv\" (UniqueName: \"kubernetes.io/projected/944c5b58-189c-401a-aeea-2d1f58baeb46-kube-api-access-d8wlv\") pod \"maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.630414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.630392    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:00.630514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.630486    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17154-brnft"]
Oct 30 20:15:00.661265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.660031    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:00.672442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.672414    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:00.702641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.701172    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:00.706558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.706533    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:00.718453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.718266    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:00.765697 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.756530    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:00.772322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.770068    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:00.802520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.795567    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:00.802520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.795815    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:00.803818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.803734    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:00.811331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.806478    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vzc\" (UniqueName: \"kubernetes.io/projected/dd6af51b-6caa-4f15-a151-dfc7928ddec8-kube-api-access-q5vzc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:00.811331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.806613    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6af51b-6caa-4f15-a151-dfc7928ddec8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:00.811331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.806647    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4cv\" (UniqueName: \"kubernetes.io/projected/e62cb1f4-31fd-4e48-9353-ec664afa53bd-kube-api-access-wj4cv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:00.811331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.806917    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62cb1f4-31fd-4e48-9353-ec664afa53bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:00.811331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.808194    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62cb1f4-31fd-4e48-9353-ec664afa53bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:00.811331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.809236    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6af51b-6caa-4f15-a151-dfc7928ddec8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:00.869060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.869028    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vzc\" (UniqueName: \"kubernetes.io/projected/dd6af51b-6caa-4f15-a151-dfc7928ddec8-kube-api-access-q5vzc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:00.886771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:00.886742    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4cv\" (UniqueName: \"kubernetes.io/projected/e62cb1f4-31fd-4e48-9353-ec664afa53bd-kube-api-access-wj4cv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:01.119198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.118805    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:01.123371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.123209    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:01.355024 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:01.354990    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f3a318_c311_494c_8152_310d7c3b35e6.slice/crio-fc5057b3d7f4f7dff1f1b425e326b36c44af0d538bd510154ef89204618aee2a WatchSource:0}: Error finding container fc5057b3d7f4f7dff1f1b425e326b36c44af0d538bd510154ef89204618aee2a: Status 404 returned error can't find the container with id fc5057b3d7f4f7dff1f1b425e326b36c44af0d538bd510154ef89204618aee2a
Oct 30 20:15:01.355489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.355450    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"]
Oct 30 20:15:01.369205 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:01.369140    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28df7940_9461_4a0d_b9c1_c74d8b4e6e33.slice/crio-c03617b028ee03589c46ec35743b016c907767e389af934ff8fa7f681db8c356 WatchSource:0}: Error finding container c03617b028ee03589c46ec35743b016c907767e389af934ff8fa7f681db8c356: Status 404 returned error can't find the container with id c03617b028ee03589c46ec35743b016c907767e389af934ff8fa7f681db8c356
Oct 30 20:15:01.369845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.369818    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"]
Oct 30 20:15:01.382096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.382058    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerStarted","Data":"fc5057b3d7f4f7dff1f1b425e326b36c44af0d538bd510154ef89204618aee2a"}
Oct 30 20:15:01.509679 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.509647    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"]
Oct 30 20:15:01.571257 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:01.571231    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14be7e43_88ac_4727_9c8d_cee1e4497ec4.slice/crio-1a3a590e8081da028bcc8c25b10cc7ac2543b6f478dfb5ba5d78e6a896275dea WatchSource:0}: Error finding container 1a3a590e8081da028bcc8c25b10cc7ac2543b6f478dfb5ba5d78e6a896275dea: Status 404 returned error can't find the container with id 1a3a590e8081da028bcc8c25b10cc7ac2543b6f478dfb5ba5d78e6a896275dea
Oct 30 20:15:01.575507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.575481    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"]
Oct 30 20:15:01.699038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.695701    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21ee149-885e-4f4f-b794-eba65e0040e5" path="/var/lib/kubelet/pods/d21ee149-885e-4f4f-b794-eba65e0040e5/volumes"
Oct 30 20:15:01.699038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.696102    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"]
Oct 30 20:15:01.702407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.701344    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"]
Oct 30 20:15:01.710575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.710513    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"]
Oct 30 20:15:01.717167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.717107    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"]
Oct 30 20:15:01.724377 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:01.723609    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a656ecb_48d2_4c7d_b1b8_5bb11cba8502.slice/crio-cc5ba0f0b1ea49d0746ed8907828f78e3100bdd1b78aa2b7172154ad1fb3d250 WatchSource:0}: Error finding container cc5ba0f0b1ea49d0746ed8907828f78e3100bdd1b78aa2b7172154ad1fb3d250: Status 404 returned error can't find the container with id cc5ba0f0b1ea49d0746ed8907828f78e3100bdd1b78aa2b7172154ad1fb3d250
Oct 30 20:15:01.726552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.726529    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"]
Oct 30 20:15:01.774795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.770480    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"]
Oct 30 20:15:01.779387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.779363    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"]
Oct 30 20:15:01.783995 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:01.783914    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62cb1f4_31fd_4e48_9353_ec664afa53bd.slice/crio-d8cf95819c4df3e43eb781761865482f33dda4cb118b97d6d094267444bb5fc2 WatchSource:0}: Error finding container d8cf95819c4df3e43eb781761865482f33dda4cb118b97d6d094267444bb5fc2: Status 404 returned error can't find the container with id d8cf95819c4df3e43eb781761865482f33dda4cb118b97d6d094267444bb5fc2
Oct 30 20:15:01.791952 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:01.791928    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6af51b_6caa_4f15_a151_dfc7928ddec8.slice/crio-262a9260b0dbc84cce1ca12010ef8814fbd48b158006367d2db782891d2f33b0 WatchSource:0}: Error finding container 262a9260b0dbc84cce1ca12010ef8814fbd48b158006367d2db782891d2f33b0: Status 404 returned error can't find the container with id 262a9260b0dbc84cce1ca12010ef8814fbd48b158006367d2db782891d2f33b0
Oct 30 20:15:01.828902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.828878    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:15:01.860684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.860655    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"]
Oct 30 20:15:01.869012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.867962    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"]
Oct 30 20:15:01.877834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.877777    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"]
Oct 30 20:15:01.881970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.881943    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-xks-ca\") pod \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") "
Oct 30 20:15:01.882105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.882072    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-openshift-ca\") pod \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") "
Oct 30 20:15:01.882230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.882213    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-kube-api-access-tjtdh\" (UniqueName: \"kubernetes.io/projected/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-kube-api-access-tjtdh\") pod \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\" (UID: \"a1c21fe7-0386-4554-bb49-63c3a43c9d8e\") "
Oct 30 20:15:01.884189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.884161    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"]
Oct 30 20:15:01.884642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.884615    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-openshift-ca" pod "a1c21fe7-0386-4554-bb49-63c3a43c9d8e" (UID: "a1c21fe7-0386-4554-bb49-63c3a43c9d8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:01.954539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.954449    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-kube-api-access-tjtdh" pod "a1c21fe7-0386-4554-bb49-63c3a43c9d8e" (UID: "a1c21fe7-0386-4554-bb49-63c3a43c9d8e"). InnerVolumeSpecName "kube-api-access-tjtdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:01.954539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.954510    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-xks-ca" pod "a1c21fe7-0386-4554-bb49-63c3a43c9d8e" (UID: "a1c21fe7-0386-4554-bb49-63c3a43c9d8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:01.955975 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:01.955946    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944c5b58_189c_401a_aeea_2d1f58baeb46.slice/crio-e2dcc124db5e975ce58479244d39e476bcce4a5f8979bca60d1d6e08e6d9a892 WatchSource:0}: Error finding container e2dcc124db5e975ce58479244d39e476bcce4a5f8979bca60d1d6e08e6d9a892: Status 404 returned error can't find the container with id e2dcc124db5e975ce58479244d39e476bcce4a5f8979bca60d1d6e08e6d9a892
Oct 30 20:15:01.984896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.984874    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:01.984896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.984896    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:01.985040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:01.984907    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjtdh\" (UniqueName: \"kubernetes.io/projected/a1c21fe7-0386-4554-bb49-63c3a43c9d8e-kube-api-access-tjtdh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:02.050187 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:02.050135    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65d56c5_518a_4275_b51f_4dd153ee64d3.slice/crio-7d579086f84f72c4ef17061f5bdae0f17747530734075cfee6b3c559587dccbc WatchSource:0}: Error finding container 7d579086f84f72c4ef17061f5bdae0f17747530734075cfee6b3c559587dccbc: Status 404 returned error can't find the container with id 7d579086f84f72c4ef17061f5bdae0f17747530734075cfee6b3c559587dccbc
Oct 30 20:15:02.301559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.301524    4519 patch_prober.go:29] interesting pod/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:15:02.301683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.301576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:15:02.301683 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:02.301612    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:15:02.393570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.393534    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" event={"ID":"944c5b58-189c-401a-aeea-2d1f58baeb46","Type":"ContainerStarted","Data":"e2dcc124db5e975ce58479244d39e476bcce4a5f8979bca60d1d6e08e6d9a892"}
Oct 30 20:15:02.395162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.395138    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8" event={"ID":"a1c21fe7-0386-4554-bb49-63c3a43c9d8e","Type":"ContainerDied","Data":"6cdcc145fc77c9dcda8cde649cb483caef8401922e84d31b1dced4f4f8b84d3e"}
Oct 30 20:15:02.395162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.395150    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"
Oct 30 20:15:02.395331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.395184    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cdcc145fc77c9dcda8cde649cb483caef8401922e84d31b1dced4f4f8b84d3e"
Oct 30 20:15:02.396954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.396924    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerStarted","Data":"c487b5669898edaf2523ea38d02482ae7f63c0d39911f7ce9523b1224f93464a"}
Oct 30 20:15:02.398360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.398338    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" event={"ID":"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502","Type":"ContainerStarted","Data":"cc5ba0f0b1ea49d0746ed8907828f78e3100bdd1b78aa2b7172154ad1fb3d250"}
Oct 30 20:15:02.400036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.400012    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" event={"ID":"5a004868-5cba-4a74-83ca-2ef0975cee10","Type":"ContainerStarted","Data":"4b0a3175d86abbbecc3237c33e6c820bc543fb4917945e17192dc3dc78f4a4be"}
Oct 30 20:15:02.400142 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.400058    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" event={"ID":"5a004868-5cba-4a74-83ca-2ef0975cee10","Type":"ContainerStarted","Data":"44bcace8fc8296d99e2498c41a30e5c89d815a136676e7cd678b9429a625ec83"}
Oct 30 20:15:02.401625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.401598    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" event={"ID":"7bb0f2b8-6560-4939-ba71-d1dc2b391a91","Type":"ContainerStarted","Data":"f5c7e5b8fdd859dea892c354d328bc774356aa7414f68f9e63f7459c43f5e098"}
Oct 30 20:15:02.403304 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.403282    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" event={"ID":"50cdcd80-a577-4bdb-8534-a60196f790e9","Type":"ContainerStarted","Data":"cf97d748dc63559de8e192ad1866f46ae4e781f1b693b9f59499444c690b9210"}
Oct 30 20:15:02.403414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.403325    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" event={"ID":"50cdcd80-a577-4bdb-8534-a60196f790e9","Type":"ContainerStarted","Data":"cc2b4c9663742f5db9549a43e62930199ef5763d09668b0ac4c4e1f3c119a3cb"}
Oct 30 20:15:02.404748 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.404723    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" event={"ID":"a4a58aaa-0c5f-4aac-af12-43e70146549a","Type":"ContainerStarted","Data":"36dd6271b0441d9a06af3f7441423e13800865b5ca15ca990131a5358905baf4"}
Oct 30 20:15:02.406553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.406273    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" event={"ID":"b65d56c5-518a-4275-b51f-4dd153ee64d3","Type":"ContainerStarted","Data":"7d579086f84f72c4ef17061f5bdae0f17747530734075cfee6b3c559587dccbc"}
Oct 30 20:15:02.409186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.409134    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" event={"ID":"a475c5c9-8cc0-4243-85b8-df9df8ba14b6","Type":"ContainerStarted","Data":"807c3c88cc88131c0c0ff15640bc5a65d1596aa1c92006fe0af0d39e702b85b9"}
Oct 30 20:15:02.410945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.410832    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" event={"ID":"fa2a5dc7-e640-4fae-abec-0fe6f439634b","Type":"ContainerStarted","Data":"fa29388dd3a01b5687038fdcb8b7b7daca1bde0d40643eee7f9cc98333dc4e77"}
Oct 30 20:15:02.412411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.412383    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" event={"ID":"dd6af51b-6caa-4f15-a151-dfc7928ddec8","Type":"ContainerStarted","Data":"262a9260b0dbc84cce1ca12010ef8814fbd48b158006367d2db782891d2f33b0"}
Oct 30 20:15:02.413822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.413801    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" event={"ID":"9964e85a-1235-4ea4-9e0c-fcfa7339aea2","Type":"ContainerStarted","Data":"31b226fd4500499d8d037717a23dcd533f7bd0f8657fe3002b77d07a6bc94d5a"}
Oct 30 20:15:02.415436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.415411    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" event={"ID":"14be7e43-88ac-4727-9c8d-cee1e4497ec4","Type":"ContainerStarted","Data":"9d80067365dccfb6504afa0934d3e130c5c10e49b67f14c5c9f2da1b30aaef7f"}
Oct 30 20:15:02.415587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.415458    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" event={"ID":"14be7e43-88ac-4727-9c8d-cee1e4497ec4","Type":"ContainerStarted","Data":"1a3a590e8081da028bcc8c25b10cc7ac2543b6f478dfb5ba5d78e6a896275dea"}
Oct 30 20:15:02.417279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.417260    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerStarted","Data":"879d552d2db1e6fd4c4fdeb95594e86c7825352fd713344989934b2d285ccc6c"}
Oct 30 20:15:02.417378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.417294    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerStarted","Data":"c03617b028ee03589c46ec35743b016c907767e389af934ff8fa7f681db8c356"}
Oct 30 20:15:02.418765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.418743    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" event={"ID":"e62cb1f4-31fd-4e48-9353-ec664afa53bd","Type":"ContainerStarted","Data":"d8cf95819c4df3e43eb781761865482f33dda4cb118b97d6d094267444bb5fc2"}
Oct 30 20:15:02.441588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.441533    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" podStartSLOduration=2.441516721 podStartE2EDuration="2.441516721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.437887203 +0000 UTC m=+2426.938590927" watchObservedRunningTime="2026-10-30 20:15:02.441516721 +0000 UTC m=+2426.942220447"
Oct 30 20:15:02.494648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.494589    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" podStartSLOduration=2.494571367 podStartE2EDuration="2.494571367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.490241434 +0000 UTC m=+2426.990945160" watchObservedRunningTime="2026-10-30 20:15:02.494571367 +0000 UTC m=+2426.995275094"
Oct 30 20:15:02.574530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.572098    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" podStartSLOduration=2.572082139 podStartE2EDuration="2.572082139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.549924696 +0000 UTC m=+2427.050628422" watchObservedRunningTime="2026-10-30 20:15:02.572082139 +0000 UTC m=+2427.072785879"
Oct 30 20:15:02.596325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.596273    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" podStartSLOduration=2.596255302 podStartE2EDuration="2.596255302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.594771366 +0000 UTC m=+2427.095475092" watchObservedRunningTime="2026-10-30 20:15:02.596255302 +0000 UTC m=+2427.096959026"
Oct 30 20:15:02.635166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:02.635080    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" podStartSLOduration=2.635066622 podStartE2EDuration="2.635066622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.632588815 +0000 UTC m=+2427.133292536" watchObservedRunningTime="2026-10-30 20:15:02.635066622 +0000 UTC m=+2427.135770345"
Oct 30 20:15:03.429317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.429285    4519 generic.go:356] "Generic (PLEG): container finished" podID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerID="879d552d2db1e6fd4c4fdeb95594e86c7825352fd713344989934b2d285ccc6c" exitCode=60
Oct 30 20:15:03.429724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.429366    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerDied","Data":"879d552d2db1e6fd4c4fdeb95594e86c7825352fd713344989934b2d285ccc6c"}
Oct 30 20:15:03.429791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.429742    4519 scope.go:122] "RemoveContainer" containerID="879d552d2db1e6fd4c4fdeb95594e86c7825352fd713344989934b2d285ccc6c"
Oct 30 20:15:03.431334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.431312    4519 generic.go:356] "Generic (PLEG): container finished" podID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerID="b6dc34b74a0df650c6ffcff2733b24abb0e4fa580545a2fa8176d2ffbd47cf38" exitCode=60
Oct 30 20:15:03.431502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.431424    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" event={"ID":"e62cb1f4-31fd-4e48-9353-ec664afa53bd","Type":"ContainerDied","Data":"b6dc34b74a0df650c6ffcff2733b24abb0e4fa580545a2fa8176d2ffbd47cf38"}
Oct 30 20:15:03.431700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.431682    4519 scope.go:122] "RemoveContainer" containerID="b6dc34b74a0df650c6ffcff2733b24abb0e4fa580545a2fa8176d2ffbd47cf38"
Oct 30 20:15:03.433231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.433208    4519 generic.go:356] "Generic (PLEG): container finished" podID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerID="5c7cc848da8db06fe66259635f5868775cc1b3302bc4bcc5654ce64a6e99abe1" exitCode=60
Oct 30 20:15:03.433352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.433312    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" event={"ID":"944c5b58-189c-401a-aeea-2d1f58baeb46","Type":"ContainerDied","Data":"5c7cc848da8db06fe66259635f5868775cc1b3302bc4bcc5654ce64a6e99abe1"}
Oct 30 20:15:03.433637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.433579    4519 scope.go:122] "RemoveContainer" containerID="5c7cc848da8db06fe66259635f5868775cc1b3302bc4bcc5654ce64a6e99abe1"
Oct 30 20:15:03.435363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.435196    4519 generic.go:356] "Generic (PLEG): container finished" podID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerID="c487b5669898edaf2523ea38d02482ae7f63c0d39911f7ce9523b1224f93464a" exitCode=60
Oct 30 20:15:03.435363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.435261    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerDied","Data":"c487b5669898edaf2523ea38d02482ae7f63c0d39911f7ce9523b1224f93464a"}
Oct 30 20:15:03.435618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.435599    4519 scope.go:122] "RemoveContainer" containerID="c487b5669898edaf2523ea38d02482ae7f63c0d39911f7ce9523b1224f93464a"
Oct 30 20:15:03.436869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.436849    4519 generic.go:356] "Generic (PLEG): container finished" podID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerID="6f47cf9b4429b3f2f28ecc7b5cd0d4244c187c1e66faa7860354a263b34636c6" exitCode=60
Oct 30 20:15:03.436977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.436922    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" event={"ID":"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502","Type":"ContainerDied","Data":"6f47cf9b4429b3f2f28ecc7b5cd0d4244c187c1e66faa7860354a263b34636c6"}
Oct 30 20:15:03.437169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.437148    4519 scope.go:122] "RemoveContainer" containerID="6f47cf9b4429b3f2f28ecc7b5cd0d4244c187c1e66faa7860354a263b34636c6"
Oct 30 20:15:03.438515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.438490    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" event={"ID":"5a004868-5cba-4a74-83ca-2ef0975cee10","Type":"ContainerDied","Data":"4b0a3175d86abbbecc3237c33e6c820bc543fb4917945e17192dc3dc78f4a4be"}
Oct 30 20:15:03.438620 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.438459    4519 generic.go:356] "Generic (PLEG): container finished" podID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerID="4b0a3175d86abbbecc3237c33e6c820bc543fb4917945e17192dc3dc78f4a4be" exitCode=60
Oct 30 20:15:03.438747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.438732    4519 scope.go:122] "RemoveContainer" containerID="4b0a3175d86abbbecc3237c33e6c820bc543fb4917945e17192dc3dc78f4a4be"
Oct 30 20:15:03.440399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.440375    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" event={"ID":"7bb0f2b8-6560-4939-ba71-d1dc2b391a91","Type":"ContainerStarted","Data":"5321c0107b87659598ab37ebeaa1f027b968fe3152c26c13a8fad2def4628aba"}
Oct 30 20:15:03.442253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.442234    4519 generic.go:356] "Generic (PLEG): container finished" podID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerID="cf97d748dc63559de8e192ad1866f46ae4e781f1b693b9f59499444c690b9210" exitCode=60
Oct 30 20:15:03.442374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.442300    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" event={"ID":"50cdcd80-a577-4bdb-8534-a60196f790e9","Type":"ContainerDied","Data":"cf97d748dc63559de8e192ad1866f46ae4e781f1b693b9f59499444c690b9210"}
Oct 30 20:15:03.442552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.442530    4519 scope.go:122] "RemoveContainer" containerID="cf97d748dc63559de8e192ad1866f46ae4e781f1b693b9f59499444c690b9210"
Oct 30 20:15:03.444906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.444887    4519 generic.go:356] "Generic (PLEG): container finished" podID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerID="3f41b1fa757e0c875f7bc27a9f9899e6aa61aa6fc696e7fbf6ccefadc01e3216" exitCode=60
Oct 30 20:15:03.445002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.444911    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" event={"ID":"a4a58aaa-0c5f-4aac-af12-43e70146549a","Type":"ContainerDied","Data":"3f41b1fa757e0c875f7bc27a9f9899e6aa61aa6fc696e7fbf6ccefadc01e3216"}
Oct 30 20:15:03.445236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.445218    4519 scope.go:122] "RemoveContainer" containerID="3f41b1fa757e0c875f7bc27a9f9899e6aa61aa6fc696e7fbf6ccefadc01e3216"
Oct 30 20:15:03.446729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.446708    4519 generic.go:356] "Generic (PLEG): container finished" podID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerID="36cbb460ada81be5d35ead868fc461cffbbe2fdf16e2f86b46f99567ae7ad0ef" exitCode=60
Oct 30 20:15:03.446800 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.446786    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" event={"ID":"b65d56c5-518a-4275-b51f-4dd153ee64d3","Type":"ContainerDied","Data":"36cbb460ada81be5d35ead868fc461cffbbe2fdf16e2f86b46f99567ae7ad0ef"}
Oct 30 20:15:03.447080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.447063    4519 scope.go:122] "RemoveContainer" containerID="36cbb460ada81be5d35ead868fc461cffbbe2fdf16e2f86b46f99567ae7ad0ef"
Oct 30 20:15:03.448993 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.448972    4519 generic.go:356] "Generic (PLEG): container finished" podID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerID="2c3832584e62f9034e47199bcae2caa281a21c04bb5308fcd71007e809732915" exitCode=60
Oct 30 20:15:03.449099 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.449049    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" event={"ID":"a475c5c9-8cc0-4243-85b8-df9df8ba14b6","Type":"ContainerDied","Data":"2c3832584e62f9034e47199bcae2caa281a21c04bb5308fcd71007e809732915"}
Oct 30 20:15:03.449303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.449274    4519 scope.go:122] "RemoveContainer" containerID="2c3832584e62f9034e47199bcae2caa281a21c04bb5308fcd71007e809732915"
Oct 30 20:15:03.458077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.458056    4519 generic.go:356] "Generic (PLEG): container finished" podID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerID="503af5441ec70a223cb83c502f8788fe1dbb6a459a5283fd5ba989ed760a73f5" exitCode=60
Oct 30 20:15:03.458706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.458684    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" event={"ID":"fa2a5dc7-e640-4fae-abec-0fe6f439634b","Type":"ContainerDied","Data":"503af5441ec70a223cb83c502f8788fe1dbb6a459a5283fd5ba989ed760a73f5"}
Oct 30 20:15:03.458794 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.458718    4519 scope.go:122] "RemoveContainer" containerID="503af5441ec70a223cb83c502f8788fe1dbb6a459a5283fd5ba989ed760a73f5"
Oct 30 20:15:03.462713 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.462693    4519 generic.go:356] "Generic (PLEG): container finished" podID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerID="dacbf51ceaada24bde98b3e70d85e6e7d7782e9d2bcb700885673298631e2bb4" exitCode=60
Oct 30 20:15:03.463207 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.463184    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" event={"ID":"dd6af51b-6caa-4f15-a151-dfc7928ddec8","Type":"ContainerDied","Data":"dacbf51ceaada24bde98b3e70d85e6e7d7782e9d2bcb700885673298631e2bb4"}
Oct 30 20:15:03.463571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.463554    4519 scope.go:122] "RemoveContainer" containerID="dacbf51ceaada24bde98b3e70d85e6e7d7782e9d2bcb700885673298631e2bb4"
Oct 30 20:15:03.465280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.465254    4519 generic.go:356] "Generic (PLEG): container finished" podID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerID="6199b120c5649612cd08bc69c6dc4b1ea7809d29c275a958b7db2f991322743d" exitCode=60
Oct 30 20:15:03.465394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.465377    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" event={"ID":"9964e85a-1235-4ea4-9e0c-fcfa7339aea2","Type":"ContainerDied","Data":"6199b120c5649612cd08bc69c6dc4b1ea7809d29c275a958b7db2f991322743d"}
Oct 30 20:15:03.465670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.465649    4519 scope.go:122] "RemoveContainer" containerID="6199b120c5649612cd08bc69c6dc4b1ea7809d29c275a958b7db2f991322743d"
Oct 30 20:15:03.469479 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.469436    4519 generic.go:356] "Generic (PLEG): container finished" podID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerID="9d80067365dccfb6504afa0934d3e130c5c10e49b67f14c5c9f2da1b30aaef7f" exitCode=60
Oct 30 20:15:03.469614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.469583    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" event={"ID":"14be7e43-88ac-4727-9c8d-cee1e4497ec4","Type":"ContainerDied","Data":"9d80067365dccfb6504afa0934d3e130c5c10e49b67f14c5c9f2da1b30aaef7f"}
Oct 30 20:15:03.470397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.470180    4519 scope.go:122] "RemoveContainer" containerID="9d80067365dccfb6504afa0934d3e130c5c10e49b67f14c5c9f2da1b30aaef7f"
Oct 30 20:15:03.477617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.477595    4519 generic.go:356] "Generic (PLEG): container finished" podID="d64c028c-e19b-4191-b338-9b863873c47b" containerID="e1dd10253e2cf36c600464a6ddccada56fcab741c4f4b75510866dd0c7088933" exitCode=0
Oct 30 20:15:03.477759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.477739    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" event={"ID":"d64c028c-e19b-4191-b338-9b863873c47b","Type":"ContainerDied","Data":"e1dd10253e2cf36c600464a6ddccada56fcab741c4f4b75510866dd0c7088933"}
Oct 30 20:15:03.477838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.477788    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n" event={"ID":"d64c028c-e19b-4191-b338-9b863873c47b","Type":"ContainerDied","Data":"e74c39370c467a49982371eac6ffbf76ab936cb639dc0a116c243e4505c98397"}
Oct 30 20:15:03.477838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.477815    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e74c39370c467a49982371eac6ffbf76ab936cb639dc0a116c243e4505c98397"
Oct 30 20:15:03.477838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.477832    4519 scope.go:122] "RemoveContainer" containerID="f8a69c3aa3a260f493591da645463e2fe8252be6d12a248cdefb5dd5c74ed391"
Oct 30 20:15:03.564551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.564507    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614009    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d64c028c-e19b-4191-b338-9b863873c47b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64c028c-e19b-4191-b338-9b863873c47b-istio-podinfo\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614120    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-socket\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614215    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-istio-token\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-istio-token\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614275    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-data\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614396    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-certs\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614483    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-credential-socket\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614513    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-kube-api-access-vznjf\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-kube-api-access-vznjf\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64c028c-e19b-4191-b338-9b863873c47b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64c028c-e19b-4191-b338-9b863873c47b-istiod-ca-cert\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614589    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-socket" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.614627    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-envoy\") pod \"d64c028c-e19b-4191-b338-9b863873c47b\" (UID: \"d64c028c-e19b-4191-b338-9b863873c47b\") "
Oct 30 20:15:03.615282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.615018    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-certs" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:03.615916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.615752    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.615916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.615777    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.617028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.617005    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64c028c-e19b-4191-b338-9b863873c47b-istiod-ca-cert" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:03.617451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.617386    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d64c028c-e19b-4191-b338-9b863873c47b-istio-podinfo" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:03.617723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.617701    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-credential-socket" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:03.618828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.618804    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-envoy" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:03.620106 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.620069    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-istio-token" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:03.620209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.620181    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-data" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:03.622500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.622107    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-kube-api-access-vznjf" pod "d64c028c-e19b-4191-b338-9b863873c47b" (UID: "d64c028c-e19b-4191-b338-9b863873c47b"). InnerVolumeSpecName "kube-api-access-vznjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:03.717683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.717598    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.717683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.717629    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznjf\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-kube-api-access-vznjf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.717683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.717643    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64c028c-e19b-4191-b338-9b863873c47b-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.717683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.717657    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.717683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.717672    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64c028c-e19b-4191-b338-9b863873c47b-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.717683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.717684    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64c028c-e19b-4191-b338-9b863873c47b-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:03.718070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:03.717695    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64c028c-e19b-4191-b338-9b863873c47b-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:04.046208 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.046155    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" podStartSLOduration=4.04614171 podStartE2EDuration="4.04614171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.045711367 +0000 UTC m=+2428.546415093" watchObservedRunningTime="2026-10-30 20:15:04.04614171 +0000 UTC m=+2428.546845433"
Oct 30 20:15:04.490656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.490586    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" event={"ID":"fa2a5dc7-e640-4fae-abec-0fe6f439634b","Type":"ContainerStarted","Data":"a4c5edccc3f15d74b02c8dbb790b6d7e78e3aae39cba7c51f53137937a72a84e"}
Oct 30 20:15:04.492967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.492937    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" event={"ID":"dd6af51b-6caa-4f15-a151-dfc7928ddec8","Type":"ContainerStarted","Data":"9cdbc5c891816d0cfb7be83731a10f9cfc8f89c6d5b4f0d102823de330ab175a"}
Oct 30 20:15:04.495247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.495218    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" event={"ID":"9964e85a-1235-4ea4-9e0c-fcfa7339aea2","Type":"ContainerStarted","Data":"34530db9e79c0e868092058f5855c7e51f55292afc704e62d58754113cc2a472"}
Oct 30 20:15:04.497407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.497379    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" event={"ID":"14be7e43-88ac-4727-9c8d-cee1e4497ec4","Type":"ContainerStarted","Data":"4bf2d5c92e73860f1b974691338bef5845be451c775874cfe10f26a84cf82e44"}
Oct 30 20:15:04.498927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.498902    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"
Oct 30 20:15:04.501407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.501383    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerStarted","Data":"c2bf50b905659f3d184a70614b2fdc182128756e6adbdbca376da191fdaf9a68"}
Oct 30 20:15:04.504167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.504142    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" event={"ID":"e62cb1f4-31fd-4e48-9353-ec664afa53bd","Type":"ContainerStarted","Data":"695e114eb9df1af9d2b0a854418143060b92a6e52b0a869b7023b6ab6f5e0558"}
Oct 30 20:15:04.507929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.507900    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" event={"ID":"944c5b58-189c-401a-aeea-2d1f58baeb46","Type":"ContainerStarted","Data":"600ec7119f83bae8ce429a9c838e830a0f8c375d0ccc472c546ffebdb18fdfba"}
Oct 30 20:15:04.510264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.510204    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerStarted","Data":"c7b1fe622322bc8a448c868bffdce7a1826ff6c4e7aaaf9b7b67d8bb92852105"}
Oct 30 20:15:04.512525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.512461    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" event={"ID":"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502","Type":"ContainerStarted","Data":"2545a47567ab9f71d22ac1c92ad17a1b0d7e1d4764ef9969d890648bc89765cc"}
Oct 30 20:15:04.514758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.514735    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" event={"ID":"5a004868-5cba-4a74-83ca-2ef0975cee10","Type":"ContainerStarted","Data":"ce478b1fc239622c5ce385aa4a5199cb54e39eb08296d9e4c4565c04afd9db43"}
Oct 30 20:15:04.516553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.516527    4519 generic.go:356] "Generic (PLEG): container finished" podID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerID="5321c0107b87659598ab37ebeaa1f027b968fe3152c26c13a8fad2def4628aba" exitCode=60
Oct 30 20:15:04.516643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.516598    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" event={"ID":"7bb0f2b8-6560-4939-ba71-d1dc2b391a91","Type":"ContainerDied","Data":"5321c0107b87659598ab37ebeaa1f027b968fe3152c26c13a8fad2def4628aba"}
Oct 30 20:15:04.516822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.516803    4519 scope.go:122] "RemoveContainer" containerID="5321c0107b87659598ab37ebeaa1f027b968fe3152c26c13a8fad2def4628aba"
Oct 30 20:15:04.520363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.520313    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" event={"ID":"50cdcd80-a577-4bdb-8534-a60196f790e9","Type":"ContainerStarted","Data":"540e70012fe681746ac512113ace2323210face05951984e7014a08f8084eebe"}
Oct 30 20:15:04.523834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.523811    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" event={"ID":"a4a58aaa-0c5f-4aac-af12-43e70146549a","Type":"ContainerStarted","Data":"18d3589b826930c18b14572c31ca1780e3f55582bc9cb28dabc7d8581d252cf1"}
Oct 30 20:15:04.526771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.526748    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" event={"ID":"b65d56c5-518a-4275-b51f-4dd153ee64d3","Type":"ContainerStarted","Data":"46dc230b6144613be192ab843c33442bf6086133f4d4eb12199ed2eead7f69ac"}
Oct 30 20:15:04.528840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.528810    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" event={"ID":"a475c5c9-8cc0-4243-85b8-df9df8ba14b6","Type":"ContainerStarted","Data":"56e62ff816c318162e55ab897ebfc14328a0ecb883e39de0b23105b6ef7a2943"}
Oct 30 20:15:04.539247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.539205    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" podStartSLOduration=4.539193024 podStartE2EDuration="4.539193024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.533850093 +0000 UTC m=+2429.034553820" watchObservedRunningTime="2026-10-30 20:15:04.539193024 +0000 UTC m=+2429.039896779"
Oct 30 20:15:04.686128 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.686064    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" podStartSLOduration=4.686041716 podStartE2EDuration="4.686041716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.666302569 +0000 UTC m=+2429.167006295" watchObservedRunningTime="2026-10-30 20:15:04.686041716 +0000 UTC m=+2429.186745441"
Oct 30 20:15:04.812568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.812521    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" podStartSLOduration=4.812506947 podStartE2EDuration="4.812506947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.771530472 +0000 UTC m=+2429.272234190" watchObservedRunningTime="2026-10-30 20:15:04.812506947 +0000 UTC m=+2429.313210664"
Oct 30 20:15:04.816337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.816303    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" podStartSLOduration=4.81629031 podStartE2EDuration="4.81629031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.811418417 +0000 UTC m=+2429.312122142" watchObservedRunningTime="2026-10-30 20:15:04.81629031 +0000 UTC m=+2429.316994033"
Oct 30 20:15:04.906255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.906107    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" podStartSLOduration=4.906062996 podStartE2EDuration="4.906062996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.858812097 +0000 UTC m=+2429.359515823" watchObservedRunningTime="2026-10-30 20:15:04.906062996 +0000 UTC m=+2429.406766719"
Oct 30 20:15:04.908688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.908540    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" podStartSLOduration=4.908493278 podStartE2EDuration="4.908493278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.8977985 +0000 UTC m=+2429.398502225" watchObservedRunningTime="2026-10-30 20:15:04.908493278 +0000 UTC m=+2429.409197002"
Oct 30 20:15:04.941360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:04.941304    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" podStartSLOduration=4.941288728 podStartE2EDuration="4.941288728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.93764511 +0000 UTC m=+2429.438348834" watchObservedRunningTime="2026-10-30 20:15:04.941288728 +0000 UTC m=+2429.441992452"
Oct 30 20:15:05.091128 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.091092    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"]
Oct 30 20:15:05.095983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.095954    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7b5635-openshift-default-5b6b96c779-dkl6n"]
Oct 30 20:15:05.138872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.138817    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" podStartSLOduration=5.138799913 podStartE2EDuration="5.138799913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:05.133198619 +0000 UTC m=+2429.633902344" watchObservedRunningTime="2026-10-30 20:15:05.138799913 +0000 UTC m=+2429.639503638"
Oct 30 20:15:05.217452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.217391    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" podStartSLOduration=5.21737292 podStartE2EDuration="5.21737292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:05.209461728 +0000 UTC m=+2429.710165451" watchObservedRunningTime="2026-10-30 20:15:05.21737292 +0000 UTC m=+2429.718076647"
Oct 30 20:15:05.542196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.542166    4519 generic.go:356] "Generic (PLEG): container finished" podID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerID="9cdbc5c891816d0cfb7be83731a10f9cfc8f89c6d5b4f0d102823de330ab175a" exitCode=60
Oct 30 20:15:05.542669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.542254    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" event={"ID":"dd6af51b-6caa-4f15-a151-dfc7928ddec8","Type":"ContainerDied","Data":"9cdbc5c891816d0cfb7be83731a10f9cfc8f89c6d5b4f0d102823de330ab175a"}
Oct 30 20:15:05.542669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.542377    4519 scope.go:122] "RemoveContainer" containerID="dacbf51ceaada24bde98b3e70d85e6e7d7782e9d2bcb700885673298631e2bb4"
Oct 30 20:15:05.542898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.542876    4519 scope.go:122] "RemoveContainer" containerID="9cdbc5c891816d0cfb7be83731a10f9cfc8f89c6d5b4f0d102823de330ab175a"
Oct 30 20:15:05.543310 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.543285    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr_odh-ai-gateway-infra(dd6af51b-6caa-4f15-a151-dfc7928ddec8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8"
Oct 30 20:15:05.544618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.544597    4519 generic.go:356] "Generic (PLEG): container finished" podID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerID="34530db9e79c0e868092058f5855c7e51f55292afc704e62d58754113cc2a472" exitCode=60
Oct 30 20:15:05.544705 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.544671    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" event={"ID":"9964e85a-1235-4ea4-9e0c-fcfa7339aea2","Type":"ContainerDied","Data":"34530db9e79c0e868092058f5855c7e51f55292afc704e62d58754113cc2a472"}
Oct 30 20:15:05.545029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.545014    4519 scope.go:122] "RemoveContainer" containerID="34530db9e79c0e868092058f5855c7e51f55292afc704e62d58754113cc2a472"
Oct 30 20:15:05.545347 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.545328    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84_odh-ai-gateway-infra(9964e85a-1235-4ea4-9e0c-fcfa7339aea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2"
Oct 30 20:15:05.547589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.547564    4519 generic.go:356] "Generic (PLEG): container finished" podID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerID="4bf2d5c92e73860f1b974691338bef5845be451c775874cfe10f26a84cf82e44" exitCode=60
Oct 30 20:15:05.547715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.547652    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" event={"ID":"14be7e43-88ac-4727-9c8d-cee1e4497ec4","Type":"ContainerDied","Data":"4bf2d5c92e73860f1b974691338bef5845be451c775874cfe10f26a84cf82e44"}
Oct 30 20:15:05.548939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.548912    4519 scope.go:122] "RemoveContainer" containerID="4bf2d5c92e73860f1b974691338bef5845be451c775874cfe10f26a84cf82e44"
Oct 30 20:15:05.549271 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.549243    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm_odh-ai-gateway-infra(14be7e43-88ac-4727-9c8d-cee1e4497ec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4"
Oct 30 20:15:05.550276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.550254    4519 generic.go:356] "Generic (PLEG): container finished" podID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerID="c2bf50b905659f3d184a70614b2fdc182128756e6adbdbca376da191fdaf9a68" exitCode=60
Oct 30 20:15:05.550366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.550333    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerDied","Data":"c2bf50b905659f3d184a70614b2fdc182128756e6adbdbca376da191fdaf9a68"}
Oct 30 20:15:05.550618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.550598    4519 scope.go:122] "RemoveContainer" containerID="c2bf50b905659f3d184a70614b2fdc182128756e6adbdbca376da191fdaf9a68"
Oct 30 20:15:05.550869 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.550843    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6_odh-ai-gateway-infra(28df7940-9461-4a0d-b9c1-c74d8b4e6e33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33"
Oct 30 20:15:05.552732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.552697    4519 generic.go:356] "Generic (PLEG): container finished" podID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerID="695e114eb9df1af9d2b0a854418143060b92a6e52b0a869b7023b6ab6f5e0558" exitCode=60
Oct 30 20:15:05.552834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.552812    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" event={"ID":"e62cb1f4-31fd-4e48-9353-ec664afa53bd","Type":"ContainerDied","Data":"695e114eb9df1af9d2b0a854418143060b92a6e52b0a869b7023b6ab6f5e0558"}
Oct 30 20:15:05.553004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.552984    4519 scope.go:122] "RemoveContainer" containerID="695e114eb9df1af9d2b0a854418143060b92a6e52b0a869b7023b6ab6f5e0558"
Oct 30 20:15:05.553260 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.553233    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv_odh-ai-gateway-infra(e62cb1f4-31fd-4e48-9353-ec664afa53bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd"
Oct 30 20:15:05.555082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.555059    4519 generic.go:356] "Generic (PLEG): container finished" podID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerID="600ec7119f83bae8ce429a9c838e830a0f8c375d0ccc472c546ffebdb18fdfba" exitCode=60
Oct 30 20:15:05.555185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.555152    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" event={"ID":"944c5b58-189c-401a-aeea-2d1f58baeb46","Type":"ContainerDied","Data":"600ec7119f83bae8ce429a9c838e830a0f8c375d0ccc472c546ffebdb18fdfba"}
Oct 30 20:15:05.555419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.555391    4519 scope.go:122] "RemoveContainer" containerID="600ec7119f83bae8ce429a9c838e830a0f8c375d0ccc472c546ffebdb18fdfba"
Oct 30 20:15:05.555666 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.555632    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k_odh-ai-gateway-infra(944c5b58-189c-401a-aeea-2d1f58baeb46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46"
Oct 30 20:15:05.561837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.561817    4519 scope.go:122] "RemoveContainer" containerID="6199b120c5649612cd08bc69c6dc4b1ea7809d29c275a958b7db2f991322743d"
Oct 30 20:15:05.564004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.563041    4519 generic.go:356] "Generic (PLEG): container finished" podID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerID="c7b1fe622322bc8a448c868bffdce7a1826ff6c4e7aaaf9b7b67d8bb92852105" exitCode=60
Oct 30 20:15:05.564004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.563122    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerDied","Data":"c7b1fe622322bc8a448c868bffdce7a1826ff6c4e7aaaf9b7b67d8bb92852105"}
Oct 30 20:15:05.564004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.563421    4519 scope.go:122] "RemoveContainer" containerID="c7b1fe622322bc8a448c868bffdce7a1826ff6c4e7aaaf9b7b67d8bb92852105"
Oct 30 20:15:05.564004 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.563678    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74_odh-ai-gateway-infra(a0f3a318-c311-494c-8152-310d7c3b35e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6"
Oct 30 20:15:05.568199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.568173    4519 generic.go:356] "Generic (PLEG): container finished" podID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerID="2545a47567ab9f71d22ac1c92ad17a1b0d7e1d4764ef9969d890648bc89765cc" exitCode=60
Oct 30 20:15:05.568315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.568294    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" event={"ID":"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502","Type":"ContainerDied","Data":"2545a47567ab9f71d22ac1c92ad17a1b0d7e1d4764ef9969d890648bc89765cc"}
Oct 30 20:15:05.568567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.568546    4519 scope.go:122] "RemoveContainer" containerID="2545a47567ab9f71d22ac1c92ad17a1b0d7e1d4764ef9969d890648bc89765cc"
Oct 30 20:15:05.568978 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.568957    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss_odh-ai-gateway-infra(9a656ecb-48d2-4c7d-b1b8-5bb11cba8502)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502"
Oct 30 20:15:05.570660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.570635    4519 generic.go:356] "Generic (PLEG): container finished" podID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerID="ce478b1fc239622c5ce385aa4a5199cb54e39eb08296d9e4c4565c04afd9db43" exitCode=60
Oct 30 20:15:05.570766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.570742    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" event={"ID":"5a004868-5cba-4a74-83ca-2ef0975cee10","Type":"ContainerDied","Data":"ce478b1fc239622c5ce385aa4a5199cb54e39eb08296d9e4c4565c04afd9db43"}
Oct 30 20:15:05.571029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.571010    4519 scope.go:122] "RemoveContainer" containerID="ce478b1fc239622c5ce385aa4a5199cb54e39eb08296d9e4c4565c04afd9db43"
Oct 30 20:15:05.571250 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.571223    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8_odh-ai-gateway-infra(5a004868-5cba-4a74-83ca-2ef0975cee10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10"
Oct 30 20:15:05.573051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.573029    4519 generic.go:356] "Generic (PLEG): container finished" podID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerID="540e70012fe681746ac512113ace2323210face05951984e7014a08f8084eebe" exitCode=60
Oct 30 20:15:05.573171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.573109    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" event={"ID":"50cdcd80-a577-4bdb-8534-a60196f790e9","Type":"ContainerDied","Data":"540e70012fe681746ac512113ace2323210face05951984e7014a08f8084eebe"}
Oct 30 20:15:05.573292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.573272    4519 scope.go:122] "RemoveContainer" containerID="540e70012fe681746ac512113ace2323210face05951984e7014a08f8084eebe"
Oct 30 20:15:05.573564 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.573517    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx_odh-ai-gateway-infra(50cdcd80-a577-4bdb-8534-a60196f790e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9"
Oct 30 20:15:05.575438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.575413    4519 generic.go:356] "Generic (PLEG): container finished" podID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerID="18d3589b826930c18b14572c31ca1780e3f55582bc9cb28dabc7d8581d252cf1" exitCode=60
Oct 30 20:15:05.575582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.575490    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" event={"ID":"a4a58aaa-0c5f-4aac-af12-43e70146549a","Type":"ContainerDied","Data":"18d3589b826930c18b14572c31ca1780e3f55582bc9cb28dabc7d8581d252cf1"}
Oct 30 20:15:05.575777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.575757    4519 scope.go:122] "RemoveContainer" containerID="18d3589b826930c18b14572c31ca1780e3f55582bc9cb28dabc7d8581d252cf1"
Oct 30 20:15:05.576046 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.576020    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k_odh-ai-gateway-infra(a4a58aaa-0c5f-4aac-af12-43e70146549a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a"
Oct 30 20:15:05.577787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.577762    4519 generic.go:356] "Generic (PLEG): container finished" podID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerID="46dc230b6144613be192ab843c33442bf6086133f4d4eb12199ed2eead7f69ac" exitCode=60
Oct 30 20:15:05.577879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.577850    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" event={"ID":"b65d56c5-518a-4275-b51f-4dd153ee64d3","Type":"ContainerDied","Data":"46dc230b6144613be192ab843c33442bf6086133f4d4eb12199ed2eead7f69ac"}
Oct 30 20:15:05.578092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.578075    4519 scope.go:122] "RemoveContainer" containerID="46dc230b6144613be192ab843c33442bf6086133f4d4eb12199ed2eead7f69ac"
Oct 30 20:15:05.578313 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.578291    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b_odh-ai-gateway-infra(b65d56c5-518a-4275-b51f-4dd153ee64d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3"
Oct 30 20:15:05.579364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.579347    4519 scope.go:122] "RemoveContainer" containerID="9d80067365dccfb6504afa0934d3e130c5c10e49b67f14c5c9f2da1b30aaef7f"
Oct 30 20:15:05.580387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.580332    4519 generic.go:356] "Generic (PLEG): container finished" podID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerID="56e62ff816c318162e55ab897ebfc14328a0ecb883e39de0b23105b6ef7a2943" exitCode=60
Oct 30 20:15:05.580539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.580454    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" event={"ID":"a475c5c9-8cc0-4243-85b8-df9df8ba14b6","Type":"ContainerDied","Data":"56e62ff816c318162e55ab897ebfc14328a0ecb883e39de0b23105b6ef7a2943"}
Oct 30 20:15:05.581185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.581162    4519 scope.go:122] "RemoveContainer" containerID="56e62ff816c318162e55ab897ebfc14328a0ecb883e39de0b23105b6ef7a2943"
Oct 30 20:15:05.581455 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.581422    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9_odh-ai-gateway-infra(a475c5c9-8cc0-4243-85b8-df9df8ba14b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6"
Oct 30 20:15:05.582906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.582883    4519 generic.go:356] "Generic (PLEG): container finished" podID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerID="a4c5edccc3f15d74b02c8dbb790b6d7e78e3aae39cba7c51f53137937a72a84e" exitCode=60
Oct 30 20:15:05.583009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.582921    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" event={"ID":"fa2a5dc7-e640-4fae-abec-0fe6f439634b","Type":"ContainerDied","Data":"a4c5edccc3f15d74b02c8dbb790b6d7e78e3aae39cba7c51f53137937a72a84e"}
Oct 30 20:15:05.583175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.583158    4519 scope.go:122] "RemoveContainer" containerID="a4c5edccc3f15d74b02c8dbb790b6d7e78e3aae39cba7c51f53137937a72a84e"
Oct 30 20:15:05.583382 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:05.583362    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm_odh-ai-gateway-infra(fa2a5dc7-e640-4fae-abec-0fe6f439634b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b"
Oct 30 20:15:05.596686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.596663    4519 scope.go:122] "RemoveContainer" containerID="879d552d2db1e6fd4c4fdeb95594e86c7825352fd713344989934b2d285ccc6c"
Oct 30 20:15:05.613493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.613453    4519 scope.go:122] "RemoveContainer" containerID="b6dc34b74a0df650c6ffcff2733b24abb0e4fa580545a2fa8176d2ffbd47cf38"
Oct 30 20:15:05.619071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.619039    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64c028c-e19b-4191-b338-9b863873c47b" path="/var/lib/kubelet/pods/d64c028c-e19b-4191-b338-9b863873c47b/volumes"
Oct 30 20:15:05.630851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.630827    4519 scope.go:122] "RemoveContainer" containerID="5c7cc848da8db06fe66259635f5868775cc1b3302bc4bcc5654ce64a6e99abe1"
Oct 30 20:15:05.647525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.647502    4519 scope.go:122] "RemoveContainer" containerID="c487b5669898edaf2523ea38d02482ae7f63c0d39911f7ce9523b1224f93464a"
Oct 30 20:15:05.666552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.666535    4519 scope.go:122] "RemoveContainer" containerID="6f47cf9b4429b3f2f28ecc7b5cd0d4244c187c1e66faa7860354a263b34636c6"
Oct 30 20:15:05.682312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.682289    4519 scope.go:122] "RemoveContainer" containerID="4b0a3175d86abbbecc3237c33e6c820bc543fb4917945e17192dc3dc78f4a4be"
Oct 30 20:15:05.696864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.696846    4519 scope.go:122] "RemoveContainer" containerID="cf97d748dc63559de8e192ad1866f46ae4e781f1b693b9f59499444c690b9210"
Oct 30 20:15:05.712501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.712480    4519 scope.go:122] "RemoveContainer" containerID="3f41b1fa757e0c875f7bc27a9f9899e6aa61aa6fc696e7fbf6ccefadc01e3216"
Oct 30 20:15:05.726617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.726593    4519 scope.go:122] "RemoveContainer" containerID="36cbb460ada81be5d35ead868fc461cffbbe2fdf16e2f86b46f99567ae7ad0ef"
Oct 30 20:15:05.742217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.742198    4519 scope.go:122] "RemoveContainer" containerID="2c3832584e62f9034e47199bcae2caa281a21c04bb5308fcd71007e809732915"
Oct 30 20:15:05.756402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:05.756378    4519 scope.go:122] "RemoveContainer" containerID="503af5441ec70a223cb83c502f8788fe1dbb6a459a5283fd5ba989ed760a73f5"
Oct 30 20:15:06.194253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.194220    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"]
Oct 30 20:15:06.921772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.921742    4519 scope.go:122] "RemoveContainer" containerID="c2bf50b905659f3d184a70614b2fdc182128756e6adbdbca376da191fdaf9a68"
Oct 30 20:15:06.928311 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.922660    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6_odh-ai-gateway-infra(28df7940-9461-4a0d-b9c1-c74d8b4e6e33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33"
Oct 30 20:15:06.932290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.932246    4519 scope.go:122] "RemoveContainer" containerID="695e114eb9df1af9d2b0a854418143060b92a6e52b0a869b7023b6ab6f5e0558"
Oct 30 20:15:06.934961 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.932674    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv_odh-ai-gateway-infra(e62cb1f4-31fd-4e48-9353-ec664afa53bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd"
Oct 30 20:15:06.936219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.936192    4519 scope.go:122] "RemoveContainer" containerID="600ec7119f83bae8ce429a9c838e830a0f8c375d0ccc472c546ffebdb18fdfba"
Oct 30 20:15:06.936671 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.936640    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k_odh-ai-gateway-infra(944c5b58-189c-401a-aeea-2d1f58baeb46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46"
Oct 30 20:15:06.938735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.938716    4519 scope.go:122] "RemoveContainer" containerID="c7b1fe622322bc8a448c868bffdce7a1826ff6c4e7aaaf9b7b67d8bb92852105"
Oct 30 20:15:06.939230 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.939206    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74_odh-ai-gateway-infra(a0f3a318-c311-494c-8152-310d7c3b35e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6"
Oct 30 20:15:06.941730 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.941712    4519 scope.go:122] "RemoveContainer" containerID="2545a47567ab9f71d22ac1c92ad17a1b0d7e1d4764ef9969d890648bc89765cc"
Oct 30 20:15:06.944687 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.944662    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss_odh-ai-gateway-infra(9a656ecb-48d2-4c7d-b1b8-5bb11cba8502)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502"
Oct 30 20:15:06.946816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.946797    4519 scope.go:122] "RemoveContainer" containerID="ce478b1fc239622c5ce385aa4a5199cb54e39eb08296d9e4c4565c04afd9db43"
Oct 30 20:15:06.947226 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.947204    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8_odh-ai-gateway-infra(5a004868-5cba-4a74-83ca-2ef0975cee10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10"
Oct 30 20:15:06.951168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.951141    4519 generic.go:356] "Generic (PLEG): container finished" podID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerID="dff2cd3f4bea88485f7fc5425e252be664e62b4f7ad435cb9f0d4940947ac111" exitCode=60
Oct 30 20:15:06.951325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.951231    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" event={"ID":"7bb0f2b8-6560-4939-ba71-d1dc2b391a91","Type":"ContainerDied","Data":"dff2cd3f4bea88485f7fc5425e252be664e62b4f7ad435cb9f0d4940947ac111"}
Oct 30 20:15:06.951325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.951294    4519 scope.go:122] "RemoveContainer" containerID="5321c0107b87659598ab37ebeaa1f027b968fe3152c26c13a8fad2def4628aba"
Oct 30 20:15:06.951766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.951740    4519 scope.go:122] "RemoveContainer" containerID="dff2cd3f4bea88485f7fc5425e252be664e62b4f7ad435cb9f0d4940947ac111"
Oct 30 20:15:06.972123 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.972087    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9_odh-ai-gateway-infra(7bb0f2b8-6560-4939-ba71-d1dc2b391a91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91"
Oct 30 20:15:06.974797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.973907    4519 scope.go:122] "RemoveContainer" containerID="540e70012fe681746ac512113ace2323210face05951984e7014a08f8084eebe"
Oct 30 20:15:06.975282 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.975258    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx_odh-ai-gateway-infra(50cdcd80-a577-4bdb-8534-a60196f790e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9"
Oct 30 20:15:06.982485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.977648    4519 scope.go:122] "RemoveContainer" containerID="18d3589b826930c18b14572c31ca1780e3f55582bc9cb28dabc7d8581d252cf1"
Oct 30 20:15:06.982970 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.982945    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k_odh-ai-gateway-infra(a4a58aaa-0c5f-4aac-af12-43e70146549a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a"
Oct 30 20:15:06.983108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.983092    4519 scope.go:122] "RemoveContainer" containerID="46dc230b6144613be192ab843c33442bf6086133f4d4eb12199ed2eead7f69ac"
Oct 30 20:15:06.983527 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.983506    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b_odh-ai-gateway-infra(b65d56c5-518a-4275-b51f-4dd153ee64d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3"
Oct 30 20:15:06.987229 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.987208    4519 scope.go:122] "RemoveContainer" containerID="56e62ff816c318162e55ab897ebfc14328a0ecb883e39de0b23105b6ef7a2943"
Oct 30 20:15:06.987528 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.987503    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9_odh-ai-gateway-infra(a475c5c9-8cc0-4243-85b8-df9df8ba14b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6"
Oct 30 20:15:06.990704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.990686    4519 scope.go:122] "RemoveContainer" containerID="a4c5edccc3f15d74b02c8dbb790b6d7e78e3aae39cba7c51f53137937a72a84e"
Oct 30 20:15:06.991052 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.991029    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm_odh-ai-gateway-infra(fa2a5dc7-e640-4fae-abec-0fe6f439634b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b"
Oct 30 20:15:06.993416 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.993393    4519 scope.go:122] "RemoveContainer" containerID="9cdbc5c891816d0cfb7be83731a10f9cfc8f89c6d5b4f0d102823de330ab175a"
Oct 30 20:15:06.993803 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.993770    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr_odh-ai-gateway-infra(dd6af51b-6caa-4f15-a151-dfc7928ddec8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8"
Oct 30 20:15:06.995403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.995383    4519 scope.go:122] "RemoveContainer" containerID="34530db9e79c0e868092058f5855c7e51f55292afc704e62d58754113cc2a472"
Oct 30 20:15:06.995817 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.995793    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84_odh-ai-gateway-infra(9964e85a-1235-4ea4-9e0c-fcfa7339aea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2"
Oct 30 20:15:06.998201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:06.998179    4519 scope.go:122] "RemoveContainer" containerID="4bf2d5c92e73860f1b974691338bef5845be451c775874cfe10f26a84cf82e44"
Oct 30 20:15:06.998449 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:06.998425    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm_odh-ai-gateway-infra(14be7e43-88ac-4727-9c8d-cee1e4497ec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4"
Oct 30 20:15:07.022562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.022535    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"]
Oct 30 20:15:07.023564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.023541    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy"
Oct 30 20:15:07.023733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.023713    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy"
Oct 30 20:15:07.023823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.023740    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy"
Oct 30 20:15:07.023823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.023751    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy"
Oct 30 20:15:07.023823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.023764    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a1c21fe7-0386-4554-bb49-63c3a43c9d8e" containerName="revoke-keys"
Oct 30 20:15:07.023823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.023771    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c21fe7-0386-4554-bb49-63c3a43c9d8e" containerName="revoke-keys"
Oct 30 20:15:07.024122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.024027    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy"
Oct 30 20:15:07.024122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.024043    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64c028c-e19b-4191-b338-9b863873c47b" containerName="istio-proxy"
Oct 30 20:15:07.024122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.024053    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c21fe7-0386-4554-bb49-63c3a43c9d8e" containerName="revoke-keys"
Oct 30 20:15:07.034219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.034194    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.046449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.042329    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"]
Oct 30 20:15:07.051218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.051098    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-87ff9f\"/\"default-dockercfg-96vx5\""
Oct 30 20:15:07.051218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.051154    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-87ff9f\"/\"kube-root-ca.crt\""
Oct 30 20:15:07.051218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.051181    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-87ff9f\"/\"openshift-service-ca.crt\""
Oct 30 20:15:07.051461 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.051281    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-87ff9f\"/\"auth-test-model-87ff9f-kserve-self-signed-certs\""
Oct 30 20:15:07.111364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.111325    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-model-cache\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.111596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.111417    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-tmp-dir\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.111596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.111554    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-home\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.112808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.112779    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-dshm\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.113291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.112984    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf1e755-9c05-48de-8b31-102f7afa9e23-tls-certs\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.113291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.113024    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twt5x\" (UniqueName: \"kubernetes.io/projected/3cf1e755-9c05-48de-8b31-102f7afa9e23-kube-api-access-twt5x\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214355 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214273    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-model-cache\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214355 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214320    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-tmp-dir\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214367    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-home\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214426    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-dshm\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214547    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf1e755-9c05-48de-8b31-102f7afa9e23-tls-certs\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214575    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twt5x\" (UniqueName: \"kubernetes.io/projected/3cf1e755-9c05-48de-8b31-102f7afa9e23-kube-api-access-twt5x\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214785    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-tmp-dir\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214903    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-home\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.214923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.214913    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-model-cache\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.217452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.217429    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf1e755-9c05-48de-8b31-102f7afa9e23-tls-certs\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.220531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.220507    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"]
Oct 30 20:15:07.226506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.226441    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ebe57b-5qcfs"]
Oct 30 20:15:07.229083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.229057    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-dshm\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.236703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.236676    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twt5x\" (UniqueName: \"kubernetes.io/projected/3cf1e755-9c05-48de-8b31-102f7afa9e23-kube-api-access-twt5x\") pod \"auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") " pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.366671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.366636    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:07.654624 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.654582    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dea5e3b-5bc0-4352-92de-0554ded085b2" path="/var/lib/kubelet/pods/3dea5e3b-5bc0-4352-92de-0554ded085b2/volumes"
Oct 30 20:15:07.842843 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:07.842805    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cf1e755_9c05_48de_8b31_102f7afa9e23.slice/crio-82f99142fed68c4d67ef196e2660c855c79d11631f81a38a955e186f7e8818ef WatchSource:0}: Error finding container 82f99142fed68c4d67ef196e2660c855c79d11631f81a38a955e186f7e8818ef: Status 404 returned error can't find the container with id 82f99142fed68c4d67ef196e2660c855c79d11631f81a38a955e186f7e8818ef
Oct 30 20:15:07.845404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:07.845380    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"]
Oct 30 20:15:08.011457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.011414    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh" event={"ID":"3cf1e755-9c05-48de-8b31-102f7afa9e23","Type":"ContainerStarted","Data":"9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113"}
Oct 30 20:15:08.011457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.011480    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh" event={"ID":"3cf1e755-9c05-48de-8b31-102f7afa9e23","Type":"ContainerStarted","Data":"82f99142fed68c4d67ef196e2660c855c79d11631f81a38a955e186f7e8818ef"}
Oct 30 20:15:08.014064 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.013636    4519 scope.go:122] "RemoveContainer" containerID="dff2cd3f4bea88485f7fc5425e252be664e62b4f7ad435cb9f0d4940947ac111"
Oct 30 20:15:08.014064 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:08.013968    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9_odh-ai-gateway-infra(7bb0f2b8-6560-4939-ba71-d1dc2b391a91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91"
Oct 30 20:15:08.051279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.051042    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh" podStartSLOduration=2.051027511 podStartE2EDuration="2.051027511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:08.050713813 +0000 UTC m=+2432.551417533" watchObservedRunningTime="2026-10-30 20:15:08.051027511 +0000 UTC m=+2432.551731235"
Oct 30 20:15:08.195570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.195536    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:08.361034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.360996    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a004868-5cba-4a74-83ca-2ef0975cee10-kube-api-access-r87cm\" (UniqueName: \"kubernetes.io/projected/5a004868-5cba-4a74-83ca-2ef0975cee10-kube-api-access-r87cm\") pod \"5a004868-5cba-4a74-83ca-2ef0975cee10\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") "
Oct 30 20:15:08.361267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.361250    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a004868-5cba-4a74-83ca-2ef0975cee10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a004868-5cba-4a74-83ca-2ef0975cee10-maas-api-service-ca\") pod \"5a004868-5cba-4a74-83ca-2ef0975cee10\" (UID: \"5a004868-5cba-4a74-83ca-2ef0975cee10\") "
Oct 30 20:15:08.361701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.361675    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a004868-5cba-4a74-83ca-2ef0975cee10-maas-api-service-ca" pod "5a004868-5cba-4a74-83ca-2ef0975cee10" (UID: "5a004868-5cba-4a74-83ca-2ef0975cee10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:08.361858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.361843    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a004868-5cba-4a74-83ca-2ef0975cee10-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:08.363250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.363223    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a004868-5cba-4a74-83ca-2ef0975cee10-kube-api-access-r87cm" pod "5a004868-5cba-4a74-83ca-2ef0975cee10" (UID: "5a004868-5cba-4a74-83ca-2ef0975cee10"). InnerVolumeSpecName "kube-api-access-r87cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:08.367368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.367339    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:08.463381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.463287    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r87cm\" (UniqueName: \"kubernetes.io/projected/5a004868-5cba-4a74-83ca-2ef0975cee10-kube-api-access-r87cm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:08.881675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.881574    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"]
Oct 30 20:15:08.881842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:08.881791    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy" containerID="cri-o://dfc5c9369c8dcc2c783c11eaff4b68805e0e3bb671345a77d7cf6f17e8ba967d" gracePeriod=30
Oct 30 20:15:09.024580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:09.024543    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"
Oct 30 20:15:09.025035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:09.024581    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8" event={"ID":"5a004868-5cba-4a74-83ca-2ef0975cee10","Type":"ContainerDied","Data":"44bcace8fc8296d99e2498c41a30e5c89d815a136676e7cd678b9429a625ec83"}
Oct 30 20:15:09.025035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:09.024656    4519 scope.go:122] "RemoveContainer" containerID="ce478b1fc239622c5ce385aa4a5199cb54e39eb08296d9e4c4565c04afd9db43"
Oct 30 20:15:09.100521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:09.086258    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"]
Oct 30 20:15:09.134034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:09.123887    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ebe57b-29889855-zfnc8"]
Oct 30 20:15:09.616168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:09.616122    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10" path="/var/lib/kubelet/pods/5a004868-5cba-4a74-83ca-2ef0975cee10/volumes"
Oct 30 20:15:14.110338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.110298    4519 generic.go:356] "Generic (PLEG): container finished" podID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerID="dfc5c9369c8dcc2c783c11eaff4b68805e0e3bb671345a77d7cf6f17e8ba967d" exitCode=0
Oct 30 20:15:14.110841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.110679    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" event={"ID":"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8","Type":"ContainerDied","Data":"dfc5c9369c8dcc2c783c11eaff4b68805e0e3bb671345a77d7cf6f17e8ba967d"}
Oct 30 20:15:14.110841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.110746    4519 scope.go:122] "RemoveContainer" containerID="d8deb4449947ace196ed08c659ba81f71f59e3414568f8e277dcb10aae0c3cd4"
Oct 30 20:15:14.186766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.186721    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"]
Oct 30 20:15:14.232531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.225975    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" containerID="cri-o://4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd" gracePeriod=30
Oct 30 20:15:14.522517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.522024    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:15:14.596284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596232    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-certs\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.596555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596354    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-credential-socket\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.596555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596439    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-token\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-token\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.596555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596491    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-socket\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.596555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596546    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-kube-api-access-whhbn\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-kube-api-access-whhbn\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.596827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596682    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-credential-socket" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:14.596891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596843    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-socket" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:14.596950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.596921    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-certs" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:14.597039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.597017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-data\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.597105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.597095    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-envoy\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.597176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.597142    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istiod-ca-cert\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.597244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.597224    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-data" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:14.597303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.597251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-podinfo\") pod \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\" (UID: \"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8\") "
Oct 30 20:15:14.597760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.597735    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istiod-ca-cert" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:14.598158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.598128    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.598266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.598162    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.598266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.598177    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.598266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.598192    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.598266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.598205    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.601330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.601301    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-token" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:14.601500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.601324    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-podinfo" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:14.601500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.601348    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-kube-api-access-whhbn" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "kube-api-access-whhbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:14.601763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.601736    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-envoy" pod "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" (UID: "7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:14.699291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.699253    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.699291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.699284    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.699291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.699293    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.699291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.699301    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whhbn\" (UniqueName: \"kubernetes.io/projected/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8-kube-api-access-whhbn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:14.930173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.930141    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"]
Oct 30 20:15:14.930856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.930839    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerName="cleanup"
Oct 30 20:15:14.930908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.930858    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerName="cleanup"
Oct 30 20:15:14.930908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.930902    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy"
Oct 30 20:15:14.930908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.930907    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy"
Oct 30 20:15:14.931010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.930921    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy"
Oct 30 20:15:14.931010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.930927    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy"
Oct 30 20:15:14.931084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.931021    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerName="cleanup"
Oct 30 20:15:14.931084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.931036    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy"
Oct 30 20:15:14.931084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.931045    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" containerName="istio-proxy"
Oct 30 20:15:14.935080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.935062    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:14.952874 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.952850    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:14.962399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:14.962364    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"]
Oct 30 20:15:15.003901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.003863    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.004101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.004000    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwz5\" (UniqueName: \"kubernetes.io/projected/9c8a3875-7095-4e8a-bebe-b124eb8f0567-kube-api-access-mpwz5\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.004157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.004086    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.106020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.105986    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.106222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.106078    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwz5\" (UniqueName: \"kubernetes.io/projected/9c8a3875-7095-4e8a-bebe-b124eb8f0567-kube-api-access-mpwz5\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.106222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.106166    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.106873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.106845    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.108561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.108530    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.125977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.125952    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwz5\" (UniqueName: \"kubernetes.io/projected/9c8a3875-7095-4e8a-bebe-b124eb8f0567-kube-api-access-mpwz5\") pod \"maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.126360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.126063    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"
Oct 30 20:15:15.126360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.126119    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7" event={"ID":"7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8","Type":"ContainerDied","Data":"9b73c743430345df9f63e9d0c973c9fb0fb48457d6cd91e7fde6ab7d7e00bb4e"}
Oct 30 20:15:15.126360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.126169    4519 scope.go:122] "RemoveContainer" containerID="dfc5c9369c8dcc2c783c11eaff4b68805e0e3bb671345a77d7cf6f17e8ba967d"
Oct 30 20:15:15.183991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.183907    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"]
Oct 30 20:15:15.194007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.193977    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c17154-openshift-default-7765bbf6d9-h4kx7"]
Oct 30 20:15:15.249906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.249516    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:15.611092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.611011    4519 scope.go:122] "RemoveContainer" containerID="a4c5edccc3f15d74b02c8dbb790b6d7e78e3aae39cba7c51f53137937a72a84e"
Oct 30 20:15:15.611250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.611182    4519 scope.go:122] "RemoveContainer" containerID="540e70012fe681746ac512113ace2323210face05951984e7014a08f8084eebe"
Oct 30 20:15:15.615867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.615841    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8" path="/var/lib/kubelet/pods/7344c7d7-bb6b-4b45-bb3a-b7f1a7924ec8/volumes"
Oct 30 20:15:15.624420 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:15.624397    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8a3875_7095_4e8a_bebe_b124eb8f0567.slice/crio-3463cdcb3e1936d0300c8b72f1859fc0938a31c85d6a464b2ce1734736feb375 WatchSource:0}: Error finding container 3463cdcb3e1936d0300c8b72f1859fc0938a31c85d6a464b2ce1734736feb375: Status 404 returned error can't find the container with id 3463cdcb3e1936d0300c8b72f1859fc0938a31c85d6a464b2ce1734736feb375
Oct 30 20:15:15.625282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:15.625260    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"]
Oct 30 20:15:16.140768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:16.140687    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd" event={"ID":"9c8a3875-7095-4e8a-bebe-b124eb8f0567","Type":"ContainerStarted","Data":"3463cdcb3e1936d0300c8b72f1859fc0938a31c85d6a464b2ce1734736feb375"}
Oct 30 20:15:16.400701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:16.400620    4519 patch_prober.go:29] interesting pod/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:15:16.400701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:16.400677    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:15:16.605973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:16.605938    4519 scope.go:122] "RemoveContainer" containerID="18d3589b826930c18b14572c31ca1780e3f55582bc9cb28dabc7d8581d252cf1"
Oct 30 20:15:16.606169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:16.606053    4519 scope.go:122] "RemoveContainer" containerID="600ec7119f83bae8ce429a9c838e830a0f8c375d0ccc472c546ffebdb18fdfba"
Oct 30 20:15:16.606169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:16.606099    4519 scope.go:122] "RemoveContainer" containerID="56e62ff816c318162e55ab897ebfc14328a0ecb883e39de0b23105b6ef7a2943"
Oct 30 20:15:17.167446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.167408    4519 generic.go:356] "Generic (PLEG): container finished" podID="9c8a3875-7095-4e8a-bebe-b124eb8f0567" containerID="6a9d65c09d200d41131ba0f295f545d4db67b3f57cb39e3bd483bc0e7eb92270" exitCode=0
Oct 30 20:15:17.168310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.167749    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd" event={"ID":"9c8a3875-7095-4e8a-bebe-b124eb8f0567","Type":"ContainerDied","Data":"6a9d65c09d200d41131ba0f295f545d4db67b3f57cb39e3bd483bc0e7eb92270"}
Oct 30 20:15:17.173351 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.173326    4519 generic.go:356] "Generic (PLEG): container finished" podID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerID="0c2297f13e0f4a5a533eefb31e2af37b84750eb876e3a9b70cf9e88d63b77d03" exitCode=60
Oct 30 20:15:17.173608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.173586    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" event={"ID":"50cdcd80-a577-4bdb-8534-a60196f790e9","Type":"ContainerDied","Data":"0c2297f13e0f4a5a533eefb31e2af37b84750eb876e3a9b70cf9e88d63b77d03"}
Oct 30 20:15:17.174225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.174203    4519 scope.go:122] "RemoveContainer" containerID="540e70012fe681746ac512113ace2323210face05951984e7014a08f8084eebe"
Oct 30 20:15:17.181778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.181757    4519 scope.go:122] "RemoveContainer" containerID="0c2297f13e0f4a5a533eefb31e2af37b84750eb876e3a9b70cf9e88d63b77d03"
Oct 30 20:15:17.182344 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:17.182317    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx_odh-ai-gateway-infra(50cdcd80-a577-4bdb-8534-a60196f790e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9"
Oct 30 20:15:17.185275 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.185252    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" event={"ID":"a475c5c9-8cc0-4243-85b8-df9df8ba14b6","Type":"ContainerStarted","Data":"45115da1f99cc4df0c75819769fd832a0b0c45021d183eb049c31965942b4ef6"}
Oct 30 20:15:17.187904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.187878    4519 generic.go:356] "Generic (PLEG): container finished" podID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerID="37d390e17913b9bd5184fc8712664da1850320360dd696e89ef985934c3dae94" exitCode=60
Oct 30 20:15:17.188011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.187967    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" event={"ID":"fa2a5dc7-e640-4fae-abec-0fe6f439634b","Type":"ContainerDied","Data":"37d390e17913b9bd5184fc8712664da1850320360dd696e89ef985934c3dae94"}
Oct 30 20:15:17.188332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.188313    4519 scope.go:122] "RemoveContainer" containerID="37d390e17913b9bd5184fc8712664da1850320360dd696e89ef985934c3dae94"
Oct 30 20:15:17.188659 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:17.188637    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm_odh-ai-gateway-infra(fa2a5dc7-e640-4fae-abec-0fe6f439634b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b"
Oct 30 20:15:17.198744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.198586    4519 scope.go:122] "RemoveContainer" containerID="a4c5edccc3f15d74b02c8dbb790b6d7e78e3aae39cba7c51f53137937a72a84e"
Oct 30 20:15:17.367104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.367066    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:17.385175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.385147    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:17.606018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.605919    4519 scope.go:122] "RemoveContainer" containerID="dff2cd3f4bea88485f7fc5425e252be664e62b4f7ad435cb9f0d4940947ac111"
Oct 30 20:15:17.606213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.606159    4519 scope.go:122] "RemoveContainer" containerID="695e114eb9df1af9d2b0a854418143060b92a6e52b0a869b7023b6ab6f5e0558"
Oct 30 20:15:17.606306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.606241    4519 scope.go:122] "RemoveContainer" containerID="4bf2d5c92e73860f1b974691338bef5845be451c775874cfe10f26a84cf82e44"
Oct 30 20:15:17.606306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.606290    4519 scope.go:122] "RemoveContainer" containerID="c2bf50b905659f3d184a70614b2fdc182128756e6adbdbca376da191fdaf9a68"
Oct 30 20:15:17.607083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.606590    4519 scope.go:122] "RemoveContainer" containerID="46dc230b6144613be192ab843c33442bf6086133f4d4eb12199ed2eead7f69ac"
Oct 30 20:15:17.607083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:17.606674    4519 scope.go:122] "RemoveContainer" containerID="9cdbc5c891816d0cfb7be83731a10f9cfc8f89c6d5b4f0d102823de330ab175a"
Oct 30 20:15:18.200608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.200575    4519 generic.go:356] "Generic (PLEG): container finished" podID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerID="45115da1f99cc4df0c75819769fd832a0b0c45021d183eb049c31965942b4ef6" exitCode=60
Oct 30 20:15:18.201003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.200653    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" event={"ID":"a475c5c9-8cc0-4243-85b8-df9df8ba14b6","Type":"ContainerDied","Data":"45115da1f99cc4df0c75819769fd832a0b0c45021d183eb049c31965942b4ef6"}
Oct 30 20:15:18.201003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.200730    4519 scope.go:122] "RemoveContainer" containerID="56e62ff816c318162e55ab897ebfc14328a0ecb883e39de0b23105b6ef7a2943"
Oct 30 20:15:18.201200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.201177    4519 scope.go:122] "RemoveContainer" containerID="45115da1f99cc4df0c75819769fd832a0b0c45021d183eb049c31965942b4ef6"
Oct 30 20:15:18.201610 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:18.201587    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9_odh-ai-gateway-infra(a475c5c9-8cc0-4243-85b8-df9df8ba14b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6"
Oct 30 20:15:18.205842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.205815    4519 generic.go:356] "Generic (PLEG): container finished" podID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerID="61d3de3f5f2e3f9e6614487dca292bea45f9be1813977ac216c1dc94a3f1c575" exitCode=60
Oct 30 20:15:18.205957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.205899    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" event={"ID":"944c5b58-189c-401a-aeea-2d1f58baeb46","Type":"ContainerDied","Data":"61d3de3f5f2e3f9e6614487dca292bea45f9be1813977ac216c1dc94a3f1c575"}
Oct 30 20:15:18.206332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.206314    4519 scope.go:122] "RemoveContainer" containerID="61d3de3f5f2e3f9e6614487dca292bea45f9be1813977ac216c1dc94a3f1c575"
Oct 30 20:15:18.206738 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:18.206717    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k_odh-ai-gateway-infra(944c5b58-189c-401a-aeea-2d1f58baeb46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46"
Oct 30 20:15:18.209775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.209755    4519 generic.go:356] "Generic (PLEG): container finished" podID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerID="41700c366e5aef9c8fed6f7d0becbaff52329208c57a526066118b46f5f49e3d" exitCode=60
Oct 30 20:15:18.209861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.209828    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" event={"ID":"a4a58aaa-0c5f-4aac-af12-43e70146549a","Type":"ContainerDied","Data":"41700c366e5aef9c8fed6f7d0becbaff52329208c57a526066118b46f5f49e3d"}
Oct 30 20:15:18.211081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.210316    4519 scope.go:122] "RemoveContainer" containerID="41700c366e5aef9c8fed6f7d0becbaff52329208c57a526066118b46f5f49e3d"
Oct 30 20:15:18.211081 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:18.210663    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k_odh-ai-gateway-infra(a4a58aaa-0c5f-4aac-af12-43e70146549a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a"
Oct 30 20:15:18.217880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.217856    4519 scope.go:122] "RemoveContainer" containerID="600ec7119f83bae8ce429a9c838e830a0f8c375d0ccc472c546ffebdb18fdfba"
Oct 30 20:15:18.222427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.222388    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:15:18.236308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.236287    4519 scope.go:122] "RemoveContainer" containerID="18d3589b826930c18b14572c31ca1780e3f55582bc9cb28dabc7d8581d252cf1"
Oct 30 20:15:18.280525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.276922    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"]
Oct 30 20:15:18.310096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.309190    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"]
Oct 30 20:15:18.357570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.357485    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"]
Oct 30 20:15:18.475092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.474711    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:18.601122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.601090    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-openshift-ca\") pod \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") "
Oct 30 20:15:18.601839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.601813    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8a3875-7095-4e8a-bebe-b124eb8f0567-kube-api-access-mpwz5\" (UniqueName: \"kubernetes.io/projected/9c8a3875-7095-4e8a-bebe-b124eb8f0567-kube-api-access-mpwz5\") pod \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") "
Oct 30 20:15:18.602243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.602219    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-xks-ca\") pod \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\" (UID: \"9c8a3875-7095-4e8a-bebe-b124eb8f0567\") "
Oct 30 20:15:18.603338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.603311    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-openshift-ca" pod "9c8a3875-7095-4e8a-bebe-b124eb8f0567" (UID: "9c8a3875-7095-4e8a-bebe-b124eb8f0567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.603704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.603682    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:18.607667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.605941    4519 scope.go:122] "RemoveContainer" containerID="c7b1fe622322bc8a448c868bffdce7a1826ff6c4e7aaaf9b7b67d8bb92852105"
Oct 30 20:15:18.607667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.606828    4519 scope.go:122] "RemoveContainer" containerID="2545a47567ab9f71d22ac1c92ad17a1b0d7e1d4764ef9969d890648bc89765cc"
Oct 30 20:15:18.607667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.607555    4519 scope.go:122] "RemoveContainer" containerID="34530db9e79c0e868092058f5855c7e51f55292afc704e62d58754113cc2a472"
Oct 30 20:15:18.620909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.620885    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8a3875-7095-4e8a-bebe-b124eb8f0567-kube-api-access-mpwz5" pod "9c8a3875-7095-4e8a-bebe-b124eb8f0567" (UID: "9c8a3875-7095-4e8a-bebe-b124eb8f0567"). InnerVolumeSpecName "kube-api-access-mpwz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.639720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.639687    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-xks-ca" pod "9c8a3875-7095-4e8a-bebe-b124eb8f0567" (UID: "9c8a3875-7095-4e8a-bebe-b124eb8f0567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:18.706630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.705921    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpwz5\" (UniqueName: \"kubernetes.io/projected/9c8a3875-7095-4e8a-bebe-b124eb8f0567-kube-api-access-mpwz5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:18.706630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.705949    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8a3875-7095-4e8a-bebe-b124eb8f0567-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:18.728220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.728167    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:18.760251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.760160    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:18.806749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.806712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cdcd80-a577-4bdb-8534-a60196f790e9-kube-api-access-kr6sr\" (UniqueName: \"kubernetes.io/projected/50cdcd80-a577-4bdb-8534-a60196f790e9-kube-api-access-kr6sr\") pod \"50cdcd80-a577-4bdb-8534-a60196f790e9\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") "
Oct 30 20:15:18.807038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.807015    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cdcd80-a577-4bdb-8534-a60196f790e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cdcd80-a577-4bdb-8534-a60196f790e9-maas-api-service-ca\") pod \"50cdcd80-a577-4bdb-8534-a60196f790e9\" (UID: \"50cdcd80-a577-4bdb-8534-a60196f790e9\") "
Oct 30 20:15:18.807601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.807570    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cdcd80-a577-4bdb-8534-a60196f790e9-maas-api-service-ca" pod "50cdcd80-a577-4bdb-8534-a60196f790e9" (UID: "50cdcd80-a577-4bdb-8534-a60196f790e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.807830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.807808    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cdcd80-a577-4bdb-8534-a60196f790e9-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:18.851334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.851298    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cdcd80-a577-4bdb-8534-a60196f790e9-kube-api-access-kr6sr" pod "50cdcd80-a577-4bdb-8534-a60196f790e9" (UID: "50cdcd80-a577-4bdb-8534-a60196f790e9"). InnerVolumeSpecName "kube-api-access-kr6sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.909047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.909019    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2a5dc7-e640-4fae-abec-0fe6f439634b-kube-api-access-xrmlw\" (UniqueName: \"kubernetes.io/projected/fa2a5dc7-e640-4fae-abec-0fe6f439634b-kube-api-access-xrmlw\") pod \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") "
Oct 30 20:15:18.909201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.909078    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2a5dc7-e640-4fae-abec-0fe6f439634b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a5dc7-e640-4fae-abec-0fe6f439634b-maas-api-service-ca\") pod \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\" (UID: \"fa2a5dc7-e640-4fae-abec-0fe6f439634b\") "
Oct 30 20:15:18.909518 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.909445    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2a5dc7-e640-4fae-abec-0fe6f439634b-maas-api-service-ca" pod "fa2a5dc7-e640-4fae-abec-0fe6f439634b" (UID: "fa2a5dc7-e640-4fae-abec-0fe6f439634b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.909861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.909838    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6sr\" (UniqueName: \"kubernetes.io/projected/50cdcd80-a577-4bdb-8534-a60196f790e9-kube-api-access-kr6sr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:18.909953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.909869    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a5dc7-e640-4fae-abec-0fe6f439634b-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:18.911819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:18.911792    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2a5dc7-e640-4fae-abec-0fe6f439634b-kube-api-access-xrmlw" pod "fa2a5dc7-e640-4fae-abec-0fe6f439634b" (UID: "fa2a5dc7-e640-4fae-abec-0fe6f439634b"). InnerVolumeSpecName "kube-api-access-xrmlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.011777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.011691    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmlw\" (UniqueName: \"kubernetes.io/projected/fa2a5dc7-e640-4fae-abec-0fe6f439634b-kube-api-access-xrmlw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:19.220845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.220815    4519 generic.go:356] "Generic (PLEG): container finished" podID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerID="f6d328803e143ddca836860f78e3cd422242130c5395e0e3f0fd69a397f79e6c" exitCode=60
Oct 30 20:15:19.221252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.220893    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" event={"ID":"7bb0f2b8-6560-4939-ba71-d1dc2b391a91","Type":"ContainerDied","Data":"f6d328803e143ddca836860f78e3cd422242130c5395e0e3f0fd69a397f79e6c"}
Oct 30 20:15:19.221252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.220997    4519 scope.go:122] "RemoveContainer" containerID="dff2cd3f4bea88485f7fc5425e252be664e62b4f7ad435cb9f0d4940947ac111"
Oct 30 20:15:19.221401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.221381    4519 scope.go:122] "RemoveContainer" containerID="f6d328803e143ddca836860f78e3cd422242130c5395e0e3f0fd69a397f79e6c"
Oct 30 20:15:19.221764 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:19.221743    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9_odh-ai-gateway-infra(7bb0f2b8-6560-4939-ba71-d1dc2b391a91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91"
Oct 30 20:15:19.222713 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.222681    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx" event={"ID":"50cdcd80-a577-4bdb-8534-a60196f790e9","Type":"ContainerDied","Data":"cc2b4c9663742f5db9549a43e62930199ef5763d09668b0ac4c4e1f3c119a3cb"}
Oct 30 20:15:19.222838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.222731    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"
Oct 30 20:15:19.227005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.226981    4519 generic.go:356] "Generic (PLEG): container finished" podID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerID="9e372dd1f3f557968fccc5e0b76bb6e7b0278728b377fb2884360a1d8da9fa99" exitCode=60
Oct 30 20:15:19.227141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.227049    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" event={"ID":"b65d56c5-518a-4275-b51f-4dd153ee64d3","Type":"ContainerDied","Data":"9e372dd1f3f557968fccc5e0b76bb6e7b0278728b377fb2884360a1d8da9fa99"}
Oct 30 20:15:19.227354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.227338    4519 scope.go:122] "RemoveContainer" containerID="9e372dd1f3f557968fccc5e0b76bb6e7b0278728b377fb2884360a1d8da9fa99"
Oct 30 20:15:19.227591 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:19.227570    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b_odh-ai-gateway-infra(b65d56c5-518a-4275-b51f-4dd153ee64d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3"
Oct 30 20:15:19.230694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.230674    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"
Oct 30 20:15:19.230786 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.230674    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm" event={"ID":"fa2a5dc7-e640-4fae-abec-0fe6f439634b","Type":"ContainerDied","Data":"fa29388dd3a01b5687038fdcb8b7b7daca1bde0d40643eee7f9cc98333dc4e77"}
Oct 30 20:15:19.232780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.232754    4519 generic.go:356] "Generic (PLEG): container finished" podID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerID="39422dc157fea4b5a6e3c388d90219f1071d4c4a3894f473cd1c3fa99fc231ec" exitCode=60
Oct 30 20:15:19.232946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.232833    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" event={"ID":"dd6af51b-6caa-4f15-a151-dfc7928ddec8","Type":"ContainerDied","Data":"39422dc157fea4b5a6e3c388d90219f1071d4c4a3894f473cd1c3fa99fc231ec"}
Oct 30 20:15:19.233455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.233436    4519 scope.go:122] "RemoveContainer" containerID="39422dc157fea4b5a6e3c388d90219f1071d4c4a3894f473cd1c3fa99fc231ec"
Oct 30 20:15:19.233835 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:19.233812    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr_odh-ai-gateway-infra(dd6af51b-6caa-4f15-a151-dfc7928ddec8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8"
Oct 30 20:15:19.235019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.234996    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" event={"ID":"9964e85a-1235-4ea4-9e0c-fcfa7339aea2","Type":"ContainerStarted","Data":"83ac132df54350d5d199ff3637b835c382336885e606b9d832b5bd5d8c3a0998"}
Oct 30 20:15:19.237579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.237554    4519 generic.go:356] "Generic (PLEG): container finished" podID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerID="36881414ebdb5826e5bac713d29e82591de21f462572ce5b44275003f46b5a26" exitCode=60
Oct 30 20:15:19.237721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.237636    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" event={"ID":"14be7e43-88ac-4727-9c8d-cee1e4497ec4","Type":"ContainerDied","Data":"36881414ebdb5826e5bac713d29e82591de21f462572ce5b44275003f46b5a26"}
Oct 30 20:15:19.238093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.238000    4519 scope.go:122] "RemoveContainer" containerID="36881414ebdb5826e5bac713d29e82591de21f462572ce5b44275003f46b5a26"
Oct 30 20:15:19.238294 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:19.238246    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm_odh-ai-gateway-infra(14be7e43-88ac-4727-9c8d-cee1e4497ec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4"
Oct 30 20:15:19.239755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.239519    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"
Oct 30 20:15:19.239755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.239548    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd" event={"ID":"9c8a3875-7095-4e8a-bebe-b124eb8f0567","Type":"ContainerDied","Data":"3463cdcb3e1936d0300c8b72f1859fc0938a31c85d6a464b2ce1734736feb375"}
Oct 30 20:15:19.239755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.239582    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3463cdcb3e1936d0300c8b72f1859fc0938a31c85d6a464b2ce1734736feb375"
Oct 30 20:15:19.244702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.244665    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerStarted","Data":"617e183e1d598241781e597e7005902740271d04f76d3962ff52fe6e2c37e30d"}
Oct 30 20:15:19.246618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.246068    4519 generic.go:356] "Generic (PLEG): container finished" podID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerID="3d8a652e6eeee7e1706dabce9ca3d8ec3c8d21fed10cb21aadb59919fa5acb8d" exitCode=60
Oct 30 20:15:19.246618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.246086    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" event={"ID":"e62cb1f4-31fd-4e48-9353-ec664afa53bd","Type":"ContainerDied","Data":"3d8a652e6eeee7e1706dabce9ca3d8ec3c8d21fed10cb21aadb59919fa5acb8d"}
Oct 30 20:15:19.246618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.246348    4519 scope.go:122] "RemoveContainer" containerID="3d8a652e6eeee7e1706dabce9ca3d8ec3c8d21fed10cb21aadb59919fa5acb8d"
Oct 30 20:15:19.246618 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:19.246578    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv_odh-ai-gateway-infra(e62cb1f4-31fd-4e48-9353-ec664afa53bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd"
Oct 30 20:15:19.258532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.257883    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerStarted","Data":"f41b90c43d8f2fe0e1c9b07ffdbf9b5589b6cae046c30e582d9c9c020e88f883"}
Oct 30 20:15:19.297076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.297035    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" event={"ID":"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502","Type":"ContainerStarted","Data":"6f00ec13ee63618de61537d930ec537c990a1a1b6dc5d218e7dbd59086ff604f"}
Oct 30 20:15:19.305764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.305743    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"]
Oct 30 20:15:19.403340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.402887    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"]
Oct 30 20:15:19.460318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.459964    4519 scope.go:122] "RemoveContainer" containerID="0c2297f13e0f4a5a533eefb31e2af37b84750eb876e3a9b70cf9e88d63b77d03"
Oct 30 20:15:19.478733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.478669    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:19.487369 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.487348    4519 scope.go:122] "RemoveContainer" containerID="46dc230b6144613be192ab843c33442bf6086133f4d4eb12199ed2eead7f69ac"
Oct 30 20:15:19.531263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.530591    4519 scope.go:122] "RemoveContainer" containerID="37d390e17913b9bd5184fc8712664da1850320360dd696e89ef985934c3dae94"
Oct 30 20:15:19.551893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.551867    4519 scope.go:122] "RemoveContainer" containerID="9cdbc5c891816d0cfb7be83731a10f9cfc8f89c6d5b4f0d102823de330ab175a"
Oct 30 20:15:19.559766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.559738    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:19.591003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.590894    4519 scope.go:122] "RemoveContainer" containerID="4bf2d5c92e73860f1b974691338bef5845be451c775874cfe10f26a84cf82e44"
Oct 30 20:15:19.616332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.616306    4519 scope.go:122] "RemoveContainer" containerID="695e114eb9df1af9d2b0a854418143060b92a6e52b0a869b7023b6ab6f5e0558"
Oct 30 20:15:19.631569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.631544    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-kube-api-access-kpzrj\" (UniqueName: \"kubernetes.io/projected/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-kube-api-access-kpzrj\") pod \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") "
Oct 30 20:15:19.631847 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.631775    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-maas-api-service-ca\") pod \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\" (UID: \"a475c5c9-8cc0-4243-85b8-df9df8ba14b6\") "
Oct 30 20:15:19.632157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.632130    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-maas-api-service-ca" pod "a475c5c9-8cc0-4243-85b8-df9df8ba14b6" (UID: "a475c5c9-8cc0-4243-85b8-df9df8ba14b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.632436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.632417    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:19.633654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.633632    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-kube-api-access-kpzrj" pod "a475c5c9-8cc0-4243-85b8-df9df8ba14b6" (UID: "a475c5c9-8cc0-4243-85b8-df9df8ba14b6"). InnerVolumeSpecName "kube-api-access-kpzrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.651254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.651229    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:19.733209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.733175    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944c5b58-189c-401a-aeea-2d1f58baeb46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944c5b58-189c-401a-aeea-2d1f58baeb46-maas-api-service-ca\") pod \"944c5b58-189c-401a-aeea-2d1f58baeb46\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") "
Oct 30 20:15:19.733511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.733422    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944c5b58-189c-401a-aeea-2d1f58baeb46-kube-api-access-d8wlv\" (UniqueName: \"kubernetes.io/projected/944c5b58-189c-401a-aeea-2d1f58baeb46-kube-api-access-d8wlv\") pod \"944c5b58-189c-401a-aeea-2d1f58baeb46\" (UID: \"944c5b58-189c-401a-aeea-2d1f58baeb46\") "
Oct 30 20:15:19.733775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.733753    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944c5b58-189c-401a-aeea-2d1f58baeb46-maas-api-service-ca" pod "944c5b58-189c-401a-aeea-2d1f58baeb46" (UID: "944c5b58-189c-401a-aeea-2d1f58baeb46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.734480 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.734447    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944c5b58-189c-401a-aeea-2d1f58baeb46-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:19.734571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.734494    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzrj\" (UniqueName: \"kubernetes.io/projected/a475c5c9-8cc0-4243-85b8-df9df8ba14b6-kube-api-access-kpzrj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:19.747411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.747112    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944c5b58-189c-401a-aeea-2d1f58baeb46-kube-api-access-d8wlv" pod "944c5b58-189c-401a-aeea-2d1f58baeb46" (UID: "944c5b58-189c-401a-aeea-2d1f58baeb46"). InnerVolumeSpecName "kube-api-access-d8wlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.835188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.835132    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a58aaa-0c5f-4aac-af12-43e70146549a-kube-api-access-759hn\" (UniqueName: \"kubernetes.io/projected/a4a58aaa-0c5f-4aac-af12-43e70146549a-kube-api-access-759hn\") pod \"a4a58aaa-0c5f-4aac-af12-43e70146549a\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") "
Oct 30 20:15:19.835406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.835374    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a58aaa-0c5f-4aac-af12-43e70146549a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a58aaa-0c5f-4aac-af12-43e70146549a-maas-api-service-ca\") pod \"a4a58aaa-0c5f-4aac-af12-43e70146549a\" (UID: \"a4a58aaa-0c5f-4aac-af12-43e70146549a\") "
Oct 30 20:15:19.836065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.836039    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8wlv\" (UniqueName: \"kubernetes.io/projected/944c5b58-189c-401a-aeea-2d1f58baeb46-kube-api-access-d8wlv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:19.836208 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.836092    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a58aaa-0c5f-4aac-af12-43e70146549a-maas-api-service-ca" pod "a4a58aaa-0c5f-4aac-af12-43e70146549a" (UID: "a4a58aaa-0c5f-4aac-af12-43e70146549a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.838173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.838150    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a58aaa-0c5f-4aac-af12-43e70146549a-kube-api-access-759hn" pod "a4a58aaa-0c5f-4aac-af12-43e70146549a" (UID: "a4a58aaa-0c5f-4aac-af12-43e70146549a"). InnerVolumeSpecName "kube-api-access-759hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.890653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.890595    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"]
Oct 30 20:15:19.899683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.899636    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-58d72c-29889855-ngggm"]
Oct 30 20:15:19.974689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.940422    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a58aaa-0c5f-4aac-af12-43e70146549a-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:19.974689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:19.940480    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759hn\" (UniqueName: \"kubernetes.io/projected/a4a58aaa-0c5f-4aac-af12-43e70146549a-kube-api-access-759hn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.092749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.092657    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"]
Oct 30 20:15:20.101835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.101803    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d3c508-29889855-rv5rx"]
Oct 30 20:15:20.121772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.121730    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:15:20.246585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246553    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-socket\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246605    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-kube-api-access-dv4k9\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-kube-api-access-dv4k9\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246647    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a28953-5648-4dad-8d3e-fe423819340f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a28953-5648-4dad-8d3e-fe423819340f-istiod-ca-cert\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246838    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-data\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246838    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-socket" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246885    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-certs\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246918    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-credential-socket\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.246978    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-istio-token\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-istio-token\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.247012    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-envoy\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.247106    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-data" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.247127    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4a28953-5648-4dad-8d3e-fe423819340f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a28953-5648-4dad-8d3e-fe423819340f-istio-podinfo\") pod \"f4a28953-5648-4dad-8d3e-fe423819340f\" (UID: \"f4a28953-5648-4dad-8d3e-fe423819340f\") "
Oct 30 20:15:20.247161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.247163    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a28953-5648-4dad-8d3e-fe423819340f-istiod-ca-cert" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.247937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.247316    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-certs" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:20.247937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.247722    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-credential-socket" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:20.248425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.248395    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.248559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.248428    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a28953-5648-4dad-8d3e-fe423819340f-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.248559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.248444    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.248559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.248457    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.248559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.248498    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.249433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.249405    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-kube-api-access-dv4k9" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "kube-api-access-dv4k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.249763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.249741    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-envoy" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:20.250212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.250193    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4a28953-5648-4dad-8d3e-fe423819340f-istio-podinfo" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:20.250271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.250227    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-istio-token" pod "f4a28953-5648-4dad-8d3e-fe423819340f" (UID: "f4a28953-5648-4dad-8d3e-fe423819340f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.289127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.289091    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"]
Oct 30 20:15:20.369574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.366373    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.369574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.366414    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a28953-5648-4dad-8d3e-fe423819340f-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.369574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.366429    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a28953-5648-4dad-8d3e-fe423819340f-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.369574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.366443    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv4k9\" (UniqueName: \"kubernetes.io/projected/f4a28953-5648-4dad-8d3e-fe423819340f-kube-api-access-dv4k9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.432859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.431131    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"]
Oct 30 20:15:20.498486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.497838    4519 generic.go:356] "Generic (PLEG): container finished" podID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerID="617e183e1d598241781e597e7005902740271d04f76d3962ff52fe6e2c37e30d" exitCode=60
Oct 30 20:15:20.498486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.497968    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerDied","Data":"617e183e1d598241781e597e7005902740271d04f76d3962ff52fe6e2c37e30d"}
Oct 30 20:15:20.498486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.498025    4519 scope.go:122] "RemoveContainer" containerID="c2bf50b905659f3d184a70614b2fdc182128756e6adbdbca376da191fdaf9a68"
Oct 30 20:15:20.518549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.510649    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"]
Oct 30 20:15:20.531260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.528901    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"
Oct 30 20:15:20.531260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.530528    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k" event={"ID":"944c5b58-189c-401a-aeea-2d1f58baeb46","Type":"ContainerDied","Data":"e2dcc124db5e975ce58479244d39e476bcce4a5f8979bca60d1d6e08e6d9a892"}
Oct 30 20:15:20.555361 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.555322    4519 generic.go:356] "Generic (PLEG): container finished" podID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerID="f41b90c43d8f2fe0e1c9b07ffdbf9b5589b6cae046c30e582d9c9c020e88f883" exitCode=60
Oct 30 20:15:20.557690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.556246    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"]
Oct 30 20:15:20.557690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.556326    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerDied","Data":"f41b90c43d8f2fe0e1c9b07ffdbf9b5589b6cae046c30e582d9c9c020e88f883"}
Oct 30 20:15:20.557690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.556843    4519 scope.go:122] "RemoveContainer" containerID="f41b90c43d8f2fe0e1c9b07ffdbf9b5589b6cae046c30e582d9c9c020e88f883"
Oct 30 20:15:20.557690 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:20.557276    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74_odh-ai-gateway-infra(a0f3a318-c311-494c-8152-310d7c3b35e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6"
Oct 30 20:15:20.561313 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.561281    4519 generic.go:356] "Generic (PLEG): container finished" podID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerID="6f00ec13ee63618de61537d930ec537c990a1a1b6dc5d218e7dbd59086ff604f" exitCode=60
Oct 30 20:15:20.561505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.561418    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" event={"ID":"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502","Type":"ContainerDied","Data":"6f00ec13ee63618de61537d930ec537c990a1a1b6dc5d218e7dbd59086ff604f"}
Oct 30 20:15:20.562234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.562214    4519 scope.go:122] "RemoveContainer" containerID="6f00ec13ee63618de61537d930ec537c990a1a1b6dc5d218e7dbd59086ff604f"
Oct 30 20:15:20.564168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.564147    4519 scope.go:122] "RemoveContainer" containerID="61d3de3f5f2e3f9e6614487dca292bea45f9be1813977ac216c1dc94a3f1c575"
Oct 30 20:15:20.564755 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:20.564454    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss_odh-ai-gateway-infra(9a656ecb-48d2-4c7d-b1b8-5bb11cba8502)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502"
Oct 30 20:15:20.566881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.566857    4519 generic.go:356] "Generic (PLEG): container finished" podID="f4a28953-5648-4dad-8d3e-fe423819340f" containerID="4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd" exitCode=0
Oct 30 20:15:20.567081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.567064    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" event={"ID":"f4a28953-5648-4dad-8d3e-fe423819340f","Type":"ContainerDied","Data":"4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd"}
Oct 30 20:15:20.567202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.567188    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6" event={"ID":"f4a28953-5648-4dad-8d3e-fe423819340f","Type":"ContainerDied","Data":"b5cc4aed87b901b4cfb05197ef6e2f8d04fe5a0229d269eb957facfccd103982"}
Oct 30 20:15:20.567349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.567336    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"
Oct 30 20:15:20.578448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.578393    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"]
Oct 30 20:15:20.582284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.582261    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k" event={"ID":"a4a58aaa-0c5f-4aac-af12-43e70146549a","Type":"ContainerDied","Data":"36dd6271b0441d9a06af3f7441423e13800865b5ca15ca990131a5358905baf4"}
Oct 30 20:15:20.589508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.582365    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"
Oct 30 20:15:20.603045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.603010    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9" event={"ID":"a475c5c9-8cc0-4243-85b8-df9df8ba14b6","Type":"ContainerDied","Data":"807c3c88cc88131c0c0ff15640bc5a65d1596aa1c92006fe0af0d39e702b85b9"}
Oct 30 20:15:20.603319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.603286    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"
Oct 30 20:15:20.611742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.611721    4519 scope.go:122] "RemoveContainer" containerID="c7b1fe622322bc8a448c868bffdce7a1826ff6c4e7aaaf9b7b67d8bb92852105"
Oct 30 20:15:20.623079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.622790    4519 generic.go:356] "Generic (PLEG): container finished" podID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerID="83ac132df54350d5d199ff3637b835c382336885e606b9d832b5bd5d8c3a0998" exitCode=60
Oct 30 20:15:20.624000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.623166    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" event={"ID":"9964e85a-1235-4ea4-9e0c-fcfa7339aea2","Type":"ContainerDied","Data":"83ac132df54350d5d199ff3637b835c382336885e606b9d832b5bd5d8c3a0998"}
Oct 30 20:15:20.640038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.639964    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"]
Oct 30 20:15:20.666654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.662430    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"]
Oct 30 20:15:20.699614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.699543    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6c4dd1-29889855-hnb6k"]
Oct 30 20:15:20.717736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.717633    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"]
Oct 30 20:15:20.774673 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:20.774631    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944c5b58_189c_401a_aeea_2d1f58baeb46.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a28953_5648_4dad_8d3e_fe423819340f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda475c5c9_8cc0_4243_85b8_df9df8ba14b6.slice/crio-807c3c88cc88131c0c0ff15640bc5a65d1596aa1c92006fe0af0d39e702b85b9\": RecentStats: unable to find data in memory cache]"
Oct 30 20:15:20.806375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.806118    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"]
Oct 30 20:15:20.815974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.815866    4519 scope.go:122] "RemoveContainer" containerID="2545a47567ab9f71d22ac1c92ad17a1b0d7e1d4764ef9969d890648bc89765cc"
Oct 30 20:15:20.820765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.820740    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"]
Oct 30 20:15:20.871613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.870354    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:20.888300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.888269    4519 scope.go:122] "RemoveContainer" containerID="4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd"
Oct 30 20:15:20.890262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.890241    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ebe57b-openshift-default-6c578db46c-j64b6"]
Oct 30 20:15:20.901597 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.901571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-kube-api-access-kwmpb\" (UniqueName: \"kubernetes.io/projected/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-kube-api-access-kwmpb\") pod \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") "
Oct 30 20:15:20.901988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.901963    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-maas-api-service-ca\") pod \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\" (UID: \"28df7940-9461-4a0d-b9c1-c74d8b4e6e33\") "
Oct 30 20:15:20.903380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.903345    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-maas-api-service-ca" pod "28df7940-9461-4a0d-b9c1-c74d8b4e6e33" (UID: "28df7940-9461-4a0d-b9c1-c74d8b4e6e33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.907815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.907684    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"]
Oct 30 20:15:20.914091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.914062    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-kube-api-access-kwmpb" pod "28df7940-9461-4a0d-b9c1-c74d8b4e6e33" (UID: "28df7940-9461-4a0d-b9c1-c74d8b4e6e33"). InnerVolumeSpecName "kube-api-access-kwmpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.914359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.914339    4519 scope.go:122] "RemoveContainer" containerID="2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25"
Oct 30 20:15:20.960846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.960813    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"]
Oct 30 20:15:20.970353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.969916    4519 scope.go:122] "RemoveContainer" containerID="4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd"
Oct 30 20:15:20.971049 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:20.970443    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd\": container with ID starting with 4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd not found: ID does not exist" containerID="4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd"
Oct 30 20:15:20.971049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.970524    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd"} err="failed to get container status \"4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd\": rpc error: code = NotFound desc = could not find container \"4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd\": container with ID starting with 4647cd54274774edd0d97e9866b8243598627063bd5ce7195491e3dbe0a613bd not found: ID does not exist"
Oct 30 20:15:20.971049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.970549    4519 scope.go:122] "RemoveContainer" containerID="2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25"
Oct 30 20:15:20.971264 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:20.971082    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25\": container with ID starting with 2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25 not found: ID does not exist" containerID="2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25"
Oct 30 20:15:20.971264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.971111    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25"} err="failed to get container status \"2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25\": rpc error: code = NotFound desc = could not find container \"2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25\": container with ID starting with 2e1d9887453120ae82548883e90b07d65a35970f69d0e17c0b84a12e35ddff25 not found: ID does not exist"
Oct 30 20:15:20.971264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.971131    4519 scope.go:122] "RemoveContainer" containerID="41700c366e5aef9c8fed6f7d0becbaff52329208c57a526066118b46f5f49e3d"
Oct 30 20:15:20.971576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:20.971514    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5db436-29889855-4pm7k"]
Oct 30 20:15:21.021191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.012205    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmpb\" (UniqueName: \"kubernetes.io/projected/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-kube-api-access-kwmpb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.021191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.012247    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28df7940-9461-4a0d-b9c1-c74d8b4e6e33-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.130194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.130112    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"]
Oct 30 20:15:21.156735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.149025    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4cfcee-29889855-6kdb9"]
Oct 30 20:15:21.174570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.173049    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:21.193641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.193614    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:21.199217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.199192    4519 scope.go:122] "RemoveContainer" containerID="45115da1f99cc4df0c75819769fd832a0b0c45021d183eb049c31965942b4ef6"
Oct 30 20:15:21.199848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.199824    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:21.215864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.215836    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65d56c5-518a-4275-b51f-4dd153ee64d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d56c5-518a-4275-b51f-4dd153ee64d3-maas-api-service-ca\") pod \"b65d56c5-518a-4275-b51f-4dd153ee64d3\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") "
Oct 30 20:15:21.216231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.216058    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-maas-api-service-ca\") pod \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") "
Oct 30 20:15:21.216231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.216132    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6af51b-6caa-4f15-a151-dfc7928ddec8-kube-api-access-q5vzc\" (UniqueName: \"kubernetes.io/projected/dd6af51b-6caa-4f15-a151-dfc7928ddec8-kube-api-access-q5vzc\") pod \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") "
Oct 30 20:15:21.216231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.216217    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-kube-api-access-f8rnq\" (UniqueName: \"kubernetes.io/projected/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-kube-api-access-f8rnq\") pod \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\" (UID: \"7bb0f2b8-6560-4939-ba71-d1dc2b391a91\") "
Oct 30 20:15:21.216447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.216259    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6af51b-6caa-4f15-a151-dfc7928ddec8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6af51b-6caa-4f15-a151-dfc7928ddec8-maas-api-service-ca\") pod \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\" (UID: \"dd6af51b-6caa-4f15-a151-dfc7928ddec8\") "
Oct 30 20:15:21.216447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.216347    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65d56c5-518a-4275-b51f-4dd153ee64d3-kube-api-access-vzfvn\" (UniqueName: \"kubernetes.io/projected/b65d56c5-518a-4275-b51f-4dd153ee64d3-kube-api-access-vzfvn\") pod \"b65d56c5-518a-4275-b51f-4dd153ee64d3\" (UID: \"b65d56c5-518a-4275-b51f-4dd153ee64d3\") "
Oct 30 20:15:21.222012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.221980    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65d56c5-518a-4275-b51f-4dd153ee64d3-kube-api-access-vzfvn" pod "b65d56c5-518a-4275-b51f-4dd153ee64d3" (UID: "b65d56c5-518a-4275-b51f-4dd153ee64d3"). InnerVolumeSpecName "kube-api-access-vzfvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.222342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.222314    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65d56c5-518a-4275-b51f-4dd153ee64d3-maas-api-service-ca" pod "b65d56c5-518a-4275-b51f-4dd153ee64d3" (UID: "b65d56c5-518a-4275-b51f-4dd153ee64d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.222703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.222678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-maas-api-service-ca" pod "7bb0f2b8-6560-4939-ba71-d1dc2b391a91" (UID: "7bb0f2b8-6560-4939-ba71-d1dc2b391a91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.225529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.223559    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6af51b-6caa-4f15-a151-dfc7928ddec8-maas-api-service-ca" pod "dd6af51b-6caa-4f15-a151-dfc7928ddec8" (UID: "dd6af51b-6caa-4f15-a151-dfc7928ddec8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.226041 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.226022    4519 scope.go:122] "RemoveContainer" containerID="34530db9e79c0e868092058f5855c7e51f55292afc704e62d58754113cc2a472"
Oct 30 20:15:21.229448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.229428    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6af51b-6caa-4f15-a151-dfc7928ddec8-kube-api-access-q5vzc" pod "dd6af51b-6caa-4f15-a151-dfc7928ddec8" (UID: "dd6af51b-6caa-4f15-a151-dfc7928ddec8"). InnerVolumeSpecName "kube-api-access-q5vzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.248393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.248363    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-kube-api-access-f8rnq" pod "7bb0f2b8-6560-4939-ba71-d1dc2b391a91" (UID: "7bb0f2b8-6560-4939-ba71-d1dc2b391a91"). InnerVolumeSpecName "kube-api-access-f8rnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.253600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.253349    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:21.259949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.259880    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:21.318016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.317986    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:21.318523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.318501    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-kube-api-access-8txsg\" (UniqueName: \"kubernetes.io/projected/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-kube-api-access-8txsg\") pod \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") "
Oct 30 20:15:21.318612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.318568    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14be7e43-88ac-4727-9c8d-cee1e4497ec4-kube-api-access-2wwwz\" (UniqueName: \"kubernetes.io/projected/14be7e43-88ac-4727-9c8d-cee1e4497ec4-kube-api-access-2wwwz\") pod \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") "
Oct 30 20:15:21.318612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.318604    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14be7e43-88ac-4727-9c8d-cee1e4497ec4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14be7e43-88ac-4727-9c8d-cee1e4497ec4-maas-api-service-ca\") pod \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\" (UID: \"14be7e43-88ac-4727-9c8d-cee1e4497ec4\") "
Oct 30 20:15:21.318746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.318647    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-maas-api-service-ca\") pod \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\" (UID: \"9964e85a-1235-4ea4-9e0c-fcfa7339aea2\") "
Oct 30 20:15:21.319176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.319148    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14be7e43-88ac-4727-9c8d-cee1e4497ec4-maas-api-service-ca" pod "14be7e43-88ac-4727-9c8d-cee1e4497ec4" (UID: "14be7e43-88ac-4727-9c8d-cee1e4497ec4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.319433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.319408    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzfvn\" (UniqueName: \"kubernetes.io/projected/b65d56c5-518a-4275-b51f-4dd153ee64d3-kube-api-access-vzfvn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.319681 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.319527    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-maas-api-service-ca" pod "9964e85a-1235-4ea4-9e0c-fcfa7339aea2" (UID: "9964e85a-1235-4ea4-9e0c-fcfa7339aea2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.319799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.319741    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14be7e43-88ac-4727-9c8d-cee1e4497ec4-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.323098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.323071    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d56c5-518a-4275-b51f-4dd153ee64d3-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.323185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.323104    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.323185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.323121    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vzc\" (UniqueName: \"kubernetes.io/projected/dd6af51b-6caa-4f15-a151-dfc7928ddec8-kube-api-access-q5vzc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.323185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.323137    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8rnq\" (UniqueName: \"kubernetes.io/projected/7bb0f2b8-6560-4939-ba71-d1dc2b391a91-kube-api-access-f8rnq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.323185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.323151    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6af51b-6caa-4f15-a151-dfc7928ddec8-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.323185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.321068    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14be7e43-88ac-4727-9c8d-cee1e4497ec4-kube-api-access-2wwwz" pod "14be7e43-88ac-4727-9c8d-cee1e4497ec4" (UID: "14be7e43-88ac-4727-9c8d-cee1e4497ec4"). InnerVolumeSpecName "kube-api-access-2wwwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.323185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.321620    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-kube-api-access-8txsg" pod "9964e85a-1235-4ea4-9e0c-fcfa7339aea2" (UID: "9964e85a-1235-4ea4-9e0c-fcfa7339aea2"). InnerVolumeSpecName "kube-api-access-8txsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.425060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.424938    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62cb1f4-31fd-4e48-9353-ec664afa53bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62cb1f4-31fd-4e48-9353-ec664afa53bd-maas-api-service-ca\") pod \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") "
Oct 30 20:15:21.425060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.424996    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62cb1f4-31fd-4e48-9353-ec664afa53bd-kube-api-access-wj4cv\" (UniqueName: \"kubernetes.io/projected/e62cb1f4-31fd-4e48-9353-ec664afa53bd-kube-api-access-wj4cv\") pod \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\" (UID: \"e62cb1f4-31fd-4e48-9353-ec664afa53bd\") "
Oct 30 20:15:21.426209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.426173    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8txsg\" (UniqueName: \"kubernetes.io/projected/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-kube-api-access-8txsg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.426342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.426218    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwwz\" (UniqueName: \"kubernetes.io/projected/14be7e43-88ac-4727-9c8d-cee1e4497ec4-kube-api-access-2wwwz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.426342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.426234    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9964e85a-1235-4ea4-9e0c-fcfa7339aea2-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.426709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.426677    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62cb1f4-31fd-4e48-9353-ec664afa53bd-maas-api-service-ca" pod "e62cb1f4-31fd-4e48-9353-ec664afa53bd" (UID: "e62cb1f4-31fd-4e48-9353-ec664afa53bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.431133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.431106    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62cb1f4-31fd-4e48-9353-ec664afa53bd-kube-api-access-wj4cv" pod "e62cb1f4-31fd-4e48-9353-ec664afa53bd" (UID: "e62cb1f4-31fd-4e48-9353-ec664afa53bd"). InnerVolumeSpecName "kube-api-access-wj4cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.527683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.527644    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62cb1f4-31fd-4e48-9353-ec664afa53bd-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.527683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.527680    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4cv\" (UniqueName: \"kubernetes.io/projected/e62cb1f4-31fd-4e48-9353-ec664afa53bd-kube-api-access-wj4cv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:21.614275 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.614236    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" path="/var/lib/kubelet/pods/50cdcd80-a577-4bdb-8534-a60196f790e9/volumes"
Oct 30 20:15:21.614781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.614765    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" path="/var/lib/kubelet/pods/944c5b58-189c-401a-aeea-2d1f58baeb46/volumes"
Oct 30 20:15:21.615183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.615170    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" path="/var/lib/kubelet/pods/a475c5c9-8cc0-4243-85b8-df9df8ba14b6/volumes"
Oct 30 20:15:21.615889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.615874    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" path="/var/lib/kubelet/pods/a4a58aaa-0c5f-4aac-af12-43e70146549a/volumes"
Oct 30 20:15:21.616330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.616317    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" path="/var/lib/kubelet/pods/f4a28953-5648-4dad-8d3e-fe423819340f/volumes"
Oct 30 20:15:21.616734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.616721    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" path="/var/lib/kubelet/pods/fa2a5dc7-e640-4fae-abec-0fe6f439634b/volumes"
Oct 30 20:15:21.641200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.641180    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"
Oct 30 20:15:21.641324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.641190    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b" event={"ID":"b65d56c5-518a-4275-b51f-4dd153ee64d3","Type":"ContainerDied","Data":"7d579086f84f72c4ef17061f5bdae0f17747530734075cfee6b3c559587dccbc"}
Oct 30 20:15:21.641324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.641259    4519 scope.go:122] "RemoveContainer" containerID="9e372dd1f3f557968fccc5e0b76bb6e7b0278728b377fb2884360a1d8da9fa99"
Oct 30 20:15:21.644591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.644557    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr" event={"ID":"dd6af51b-6caa-4f15-a151-dfc7928ddec8","Type":"ContainerDied","Data":"262a9260b0dbc84cce1ca12010ef8814fbd48b158006367d2db782891d2f33b0"}
Oct 30 20:15:21.644704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.644565    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"
Oct 30 20:15:21.646025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.645999    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"
Oct 30 20:15:21.646120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.646020    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84" event={"ID":"9964e85a-1235-4ea4-9e0c-fcfa7339aea2","Type":"ContainerDied","Data":"31b226fd4500499d8d037717a23dcd533f7bd0f8657fe3002b77d07a6bc94d5a"}
Oct 30 20:15:21.647860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.647339    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm" event={"ID":"14be7e43-88ac-4727-9c8d-cee1e4497ec4","Type":"ContainerDied","Data":"1a3a590e8081da028bcc8c25b10cc7ac2543b6f478dfb5ba5d78e6a896275dea"}
Oct 30 20:15:21.647860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.647349    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"
Oct 30 20:15:21.648732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.648713    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"
Oct 30 20:15:21.648840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.648755    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6" event={"ID":"28df7940-9461-4a0d-b9c1-c74d8b4e6e33","Type":"ContainerDied","Data":"c03617b028ee03589c46ec35743b016c907767e389af934ff8fa7f681db8c356"}
Oct 30 20:15:21.650281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.650264    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"
Oct 30 20:15:21.650372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.650315    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv" event={"ID":"e62cb1f4-31fd-4e48-9353-ec664afa53bd","Type":"ContainerDied","Data":"d8cf95819c4df3e43eb781761865482f33dda4cb118b97d6d094267444bb5fc2"}
Oct 30 20:15:21.659507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.659372    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9" event={"ID":"7bb0f2b8-6560-4939-ba71-d1dc2b391a91","Type":"ContainerDied","Data":"f5c7e5b8fdd859dea892c354d328bc774356aa7414f68f9e63f7459c43f5e098"}
Oct 30 20:15:21.659507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.659452    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"
Oct 30 20:15:21.708672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.708647    4519 scope.go:122] "RemoveContainer" containerID="39422dc157fea4b5a6e3c388d90219f1071d4c4a3894f473cd1c3fa99fc231ec"
Oct 30 20:15:21.724437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.724329    4519 scope.go:122] "RemoveContainer" containerID="83ac132df54350d5d199ff3637b835c382336885e606b9d832b5bd5d8c3a0998"
Oct 30 20:15:21.761638 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.761533    4519 scope.go:122] "RemoveContainer" containerID="36881414ebdb5826e5bac713d29e82591de21f462572ce5b44275003f46b5a26"
Oct 30 20:15:21.790896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.790868    4519 scope.go:122] "RemoveContainer" containerID="617e183e1d598241781e597e7005902740271d04f76d3962ff52fe6e2c37e30d"
Oct 30 20:15:21.817652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.817540    4519 scope.go:122] "RemoveContainer" containerID="3d8a652e6eeee7e1706dabce9ca3d8ec3c8d21fed10cb21aadb59919fa5acb8d"
Oct 30 20:15:21.865009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.864984    4519 scope.go:122] "RemoveContainer" containerID="f6d328803e143ddca836860f78e3cd422242130c5395e0e3f0fd69a397f79e6c"
Oct 30 20:15:21.877203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.877158    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:21.891020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.890912    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"]
Oct 30 20:15:21.895814 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.895784    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6cd2ad-29889855-j97k6"]
Oct 30 20:15:21.919257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.919233    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:21.933990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.933956    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-kube-api-access-zjk5c\" (UniqueName: \"kubernetes.io/projected/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-kube-api-access-zjk5c\") pod \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") "
Oct 30 20:15:21.934261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.934225    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-maas-api-service-ca\") pod \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\" (UID: \"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502\") "
Oct 30 20:15:21.935584 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.935546    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-maas-api-service-ca" pod "9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" (UID: "9a656ecb-48d2-4c7d-b1b8-5bb11cba8502"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.938371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.938345    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-kube-api-access-zjk5c" pod "9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" (UID: "9a656ecb-48d2-4c7d-b1b8-5bb11cba8502"). InnerVolumeSpecName "kube-api-access-zjk5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.940094 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.940067    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"]
Oct 30 20:15:21.944735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.944713    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889855-kmpcr"]
Oct 30 20:15:21.976945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.976874    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"]
Oct 30 20:15:21.983428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:21.983402    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889855-6p57b"]
Oct 30 20:15:22.012272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.012246    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"]
Oct 30 20:15:22.016062 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.016039    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8fc40-29889855-ppc84"]
Oct 30 20:15:22.036186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.036160    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f3a318-c311-494c-8152-310d7c3b35e6-kube-api-access-dd972\" (UniqueName: \"kubernetes.io/projected/a0f3a318-c311-494c-8152-310d7c3b35e6-kube-api-access-dd972\") pod \"a0f3a318-c311-494c-8152-310d7c3b35e6\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") "
Oct 30 20:15:22.036328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.036313    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f3a318-c311-494c-8152-310d7c3b35e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f3a318-c311-494c-8152-310d7c3b35e6-maas-api-service-ca\") pod \"a0f3a318-c311-494c-8152-310d7c3b35e6\" (UID: \"a0f3a318-c311-494c-8152-310d7c3b35e6\") "
Oct 30 20:15:22.036676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.036654    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f3a318-c311-494c-8152-310d7c3b35e6-maas-api-service-ca" pod "a0f3a318-c311-494c-8152-310d7c3b35e6" (UID: "a0f3a318-c311-494c-8152-310d7c3b35e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:22.036813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.036793    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjk5c\" (UniqueName: \"kubernetes.io/projected/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-kube-api-access-zjk5c\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:22.036856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.036815    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f3a318-c311-494c-8152-310d7c3b35e6-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:22.036856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.036825    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:22.038355 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.038332    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f3a318-c311-494c-8152-310d7c3b35e6-kube-api-access-dd972" pod "a0f3a318-c311-494c-8152-310d7c3b35e6" (UID: "a0f3a318-c311-494c-8152-310d7c3b35e6"). InnerVolumeSpecName "kube-api-access-dd972". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:22.052592 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.052567    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"]
Oct 30 20:15:22.058230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.058200    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-87ff9f-29889855-l8mmm"]
Oct 30 20:15:22.085933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.085901    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"]
Oct 30 20:15:22.093936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.093906    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-159b3a-29889855-tbhpv"]
Oct 30 20:15:22.126311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.126279    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"]
Oct 30 20:15:22.130329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.130300    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889855-t45q9"]
Oct 30 20:15:22.138145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.138111    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd972\" (UniqueName: \"kubernetes.io/projected/a0f3a318-c311-494c-8152-310d7c3b35e6-kube-api-access-dd972\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:22.670926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.670891    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74" event={"ID":"a0f3a318-c311-494c-8152-310d7c3b35e6","Type":"ContainerDied","Data":"fc5057b3d7f4f7dff1f1b425e326b36c44af0d538bd510154ef89204618aee2a"}
Oct 30 20:15:22.671396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.670964    4519 scope.go:122] "RemoveContainer" containerID="f41b90c43d8f2fe0e1c9b07ffdbf9b5589b6cae046c30e582d9c9c020e88f883"
Oct 30 20:15:22.671396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.670901    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"
Oct 30 20:15:22.673095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.673074    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"
Oct 30 20:15:22.673220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.673072    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss" event={"ID":"9a656ecb-48d2-4c7d-b1b8-5bb11cba8502","Type":"ContainerDied","Data":"cc5ba0f0b1ea49d0746ed8907828f78e3100bdd1b78aa2b7172154ad1fb3d250"}
Oct 30 20:15:22.688183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.688162    4519 scope.go:122] "RemoveContainer" containerID="6f00ec13ee63618de61537d930ec537c990a1a1b6dc5d218e7dbd59086ff604f"
Oct 30 20:15:22.736819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.736790    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"]
Oct 30 20:15:22.747884 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.747856    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889855-d76ss"]
Oct 30 20:15:22.827621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.827577    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"]
Oct 30 20:15:22.833687 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:22.833656    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4a31d3-29889855-m4d74"]
Oct 30 20:15:23.613388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.613356    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" path="/var/lib/kubelet/pods/14be7e43-88ac-4727-9c8d-cee1e4497ec4/volumes"
Oct 30 20:15:23.613882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.613865    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" path="/var/lib/kubelet/pods/28df7940-9461-4a0d-b9c1-c74d8b4e6e33/volumes"
Oct 30 20:15:23.614317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.614303    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" path="/var/lib/kubelet/pods/7bb0f2b8-6560-4939-ba71-d1dc2b391a91/volumes"
Oct 30 20:15:23.615086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.615070    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" path="/var/lib/kubelet/pods/9964e85a-1235-4ea4-9e0c-fcfa7339aea2/volumes"
Oct 30 20:15:23.615540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.615521    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" path="/var/lib/kubelet/pods/9a656ecb-48d2-4c7d-b1b8-5bb11cba8502/volumes"
Oct 30 20:15:23.615959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.615945    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" path="/var/lib/kubelet/pods/a0f3a318-c311-494c-8152-310d7c3b35e6/volumes"
Oct 30 20:15:23.616616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.616602    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" path="/var/lib/kubelet/pods/b65d56c5-518a-4275-b51f-4dd153ee64d3/volumes"
Oct 30 20:15:23.617019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.616998    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" path="/var/lib/kubelet/pods/dd6af51b-6caa-4f15-a151-dfc7928ddec8/volumes"
Oct 30 20:15:23.617396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:23.617383    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" path="/var/lib/kubelet/pods/e62cb1f4-31fd-4e48-9353-ec664afa53bd/volumes"
Oct 30 20:15:32.224437 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:15:32.224383    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:15:41.264434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:41.264393    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"]
Oct 30 20:15:41.273662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:41.273630    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58d72c-5tbw8"]
Oct 30 20:15:41.613978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:41.613900    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c21fe7-0386-4554-bb49-63c3a43c9d8e" path="/var/lib/kubelet/pods/a1c21fe7-0386-4554-bb49-63c3a43c9d8e/volumes"
Oct 30 20:15:43.974340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:43.974306    4519 scope.go:122] "RemoveContainer" containerID="e1dd10253e2cf36c600464a6ddccada56fcab741c4f4b75510866dd0c7088933"
Oct 30 20:15:48.180965 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:48.180891    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"]
Oct 30 20:15:48.181952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:48.181845    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" containerID="cri-o://a237fb21281537ecef3ae72cd09d7f5e1e79c024992fc18884dbe5f12884485d" gracePeriod=30
Oct 30 20:15:49.751228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.751121    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"]
Oct 30 20:15:49.752086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752061    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9c8a3875-7095-4e8a-bebe-b124eb8f0567" containerName="revoke-keys"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752086    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8a3875-7095-4e8a-bebe-b124eb8f0567" containerName="revoke-keys"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752107    4519 cpu_manager.go:414] "containerMap: removing container" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752114    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752128    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752135    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752150    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752157    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752168    4519 cpu_manager.go:414] "containerMap: removing container" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752175    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752186    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752193    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752207    4519 cpu_manager.go:414] "containerMap: removing container" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:49.752212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752214    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752226    4519 cpu_manager.go:414] "containerMap: removing container" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752234    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752245    4519 cpu_manager.go:414] "containerMap: removing container" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752253    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752266    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752273    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752284    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752290    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752300    4519 cpu_manager.go:414] "containerMap: removing container" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752307    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752319    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752326    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752337    4519 cpu_manager.go:414] "containerMap: removing container" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752345    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752355    4519 cpu_manager.go:414] "containerMap: removing container" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752362    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752375    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752382    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752399    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752406    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752417    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752424    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.752970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752439    4519 cpu_manager.go:414] "containerMap: removing container" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.752446    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753206    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753216    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753237    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753244    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753257    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753264    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753273    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753282    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.754431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753298    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.758515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.753305    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758528    4519 cpu_manager.go:414] "containerMap: removing container" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758539    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758550    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758557    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758569    4519 cpu_manager.go:414] "containerMap: removing container" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758577    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758589    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758597    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758605    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758611    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758622    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.758627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758629    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758648    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758657    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758679    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758686    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758698    4519 cpu_manager.go:414] "containerMap: removing container" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758705    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758723    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758730    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758742    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758749    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758758    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758765    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758800    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758810    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758826    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758833    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758850    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758860    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758871    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.758878    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759057    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759071    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759084    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8a3875-7095-4e8a-bebe-b124eb8f0567" containerName="revoke-keys"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759095    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759105    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759115    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a004868-5cba-4a74-83ca-2ef0975cee10" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759127    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759136    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759147    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759157    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759166    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759176    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759185    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759193    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759203    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759212    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759248    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="28df7940-9461-4a0d-b9c1-c74d8b4e6e33" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759256    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759267    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759275    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759285    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759295    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759304    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759312    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a28953-5648-4dad-8d3e-fe423819340f" containerName="istio-proxy"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759324    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a656ecb-48d2-4c7d-b1b8-5bb11cba8502" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759346    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a58aaa-0c5f-4aac-af12-43e70146549a" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759355    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759364    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759376    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:49.759356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759385    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:49.761829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759392    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:49.761829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759401    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:49.761829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759413    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:49.761829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759425    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:49.761829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759434    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:49.761829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759442    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:49.761829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.759450    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:49.772908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.772872    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:49.785616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.784876    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"]
Oct 30 20:15:49.798894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.793376    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"kube-root-ca.crt\""
Oct 30 20:15:49.798894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.793503    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"default-dockercfg-v7r9c\""
Oct 30 20:15:49.805739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.805714    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"auth-test-model-5c2e32-kserve-self-signed-certs\""
Oct 30 20:15:49.806198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.806177    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"openshift-service-ca.crt\""
Oct 30 20:15:49.964486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.964111    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndn84\" (UniqueName: \"kubernetes.io/projected/6d959bec-39ca-41c7-b751-c146392a88ef-kube-api-access-ndn84\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:49.964486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.964309    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-home\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:49.964486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.964367    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d959bec-39ca-41c7-b751-c146392a88ef-tls-certs\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:49.964771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.964530    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-model-cache\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:49.964771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.964678    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-dshm\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:49.964771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:49.964731    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-tmp-dir\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.065994    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-dshm\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.066541    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-tmp-dir\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.066605    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndn84\" (UniqueName: \"kubernetes.io/projected/6d959bec-39ca-41c7-b751-c146392a88ef-kube-api-access-ndn84\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.067353    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-tmp-dir\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.067497    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-home\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.067546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d959bec-39ca-41c7-b751-c146392a88ef-tls-certs\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.067617    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-model-cache\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.067959    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-model-cache\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.068536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.068184    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-home\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.069170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.068934    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-dshm\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.105763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.072019    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d959bec-39ca-41c7-b751-c146392a88ef-tls-certs\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.105763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.089070    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndn84\" (UniqueName: \"kubernetes.io/projected/6d959bec-39ca-41c7-b751-c146392a88ef-kube-api-access-ndn84\") pod \"auth-test-model-5c2e32-kserve-699d6ddc89-22nt9\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.105763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.104156    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:50.787828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.787791    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"]
Oct 30 20:15:50.871731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.871698    4519 patch_prober.go:29] interesting pod/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:15:50.871909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:50.871763    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:15:51.300248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:51.300205    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9" event={"ID":"6d959bec-39ca-41c7-b751-c146392a88ef","Type":"ContainerStarted","Data":"2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0"}
Oct 30 20:15:51.300442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:51.300260    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9" event={"ID":"6d959bec-39ca-41c7-b751-c146392a88ef","Type":"ContainerStarted","Data":"b3c1b59592b06bff0c09153295934e594d6b505c0681f2d47fe061658aaaa662"}
Oct 30 20:15:51.344200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:51.344146    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9" podStartSLOduration=2.344132267 podStartE2EDuration="2.344132267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:51.339948423 +0000 UTC m=+2475.840652150" watchObservedRunningTime="2026-10-30 20:15:51.344132267 +0000 UTC m=+2475.844836038"
Oct 30 20:15:52.104979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:52.104942    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:15:53.322831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.322798    4519 generic.go:356] "Generic (PLEG): container finished" podID="db80b810-4764-4663-839e-1bb464bafcf8" containerID="a237fb21281537ecef3ae72cd09d7f5e1e79c024992fc18884dbe5f12884485d" exitCode=0
Oct 30 20:15:53.323282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.322875    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" event={"ID":"db80b810-4764-4663-839e-1bb464bafcf8","Type":"ContainerDied","Data":"a237fb21281537ecef3ae72cd09d7f5e1e79c024992fc18884dbe5f12884485d"}
Oct 30 20:15:53.323282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.322935    4519 scope.go:122] "RemoveContainer" containerID="321e539d07349d1f39cb41042023fe2899ce5778dae3df5c99e02132eb2c47d2"
Oct 30 20:15:53.496248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.496215    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.555731    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-data\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.555779    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-envoy\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.555824    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db80b810-4764-4663-839e-1bb464bafcf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db80b810-4764-4663-839e-1bb464bafcf8-istio-podinfo\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.555892    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db80b810-4764-4663-839e-1bb464bafcf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db80b810-4764-4663-839e-1bb464bafcf8-istiod-ca-cert\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.555933    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-credential-socket\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.555963    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-socket\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.556062    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-kube-api-access-s82sg\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-kube-api-access-s82sg\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.556150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-certs\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.556393    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-istio-token\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-istio-token\") pod \"db80b810-4764-4663-839e-1bb464bafcf8\" (UID: \"db80b810-4764-4663-839e-1bb464bafcf8\") "
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.556397    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-data" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:53.556778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.556719    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-socket" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:53.557566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.557135    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.557566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.557154    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.558289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.557862    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db80b810-4764-4663-839e-1bb464bafcf8-istiod-ca-cert" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:53.558767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.558506    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-credential-socket" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:53.559186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.559157    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-certs" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:53.560990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.560950    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db80b810-4764-4663-839e-1bb464bafcf8-istio-podinfo" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:53.560990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.560977    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-istio-token" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:53.561686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.561654    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-envoy" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:53.561933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.561909    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-kube-api-access-s82sg" pod "db80b810-4764-4663-839e-1bb464bafcf8" (UID: "db80b810-4764-4663-839e-1bb464bafcf8"). InnerVolumeSpecName "kube-api-access-s82sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:53.658612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.658569    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.658612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.658603    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.658612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.658615    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db80b810-4764-4663-839e-1bb464bafcf8-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.658908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.658624    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db80b810-4764-4663-839e-1bb464bafcf8-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.658908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.658633    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.658908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.658643    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82sg\" (UniqueName: \"kubernetes.io/projected/db80b810-4764-4663-839e-1bb464bafcf8-kube-api-access-s82sg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:53.658908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:53.658652    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db80b810-4764-4663-839e-1bb464bafcf8-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:15:54.334158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:54.334128    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"
Oct 30 20:15:54.334664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:54.334158    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96" event={"ID":"db80b810-4764-4663-839e-1bb464bafcf8","Type":"ContainerDied","Data":"719fc88beb45ece155833256808452148ef8f2b6546c5c4fdd460fcbf053b91a"}
Oct 30 20:15:54.334664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:54.334220    4519 scope.go:122] "RemoveContainer" containerID="a237fb21281537ecef3ae72cd09d7f5e1e79c024992fc18884dbe5f12884485d"
Oct 30 20:15:54.401528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:54.401458    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"]
Oct 30 20:15:54.408108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:54.408070    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-58d72c-openshift-default-56ff58c7f5-jfr96"]
Oct 30 20:15:55.623752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:55.623709    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db80b810-4764-4663-839e-1bb464bafcf8" path="/var/lib/kubelet/pods/db80b810-4764-4663-839e-1bb464bafcf8/volumes"
Oct 30 20:15:56.609876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.609840    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"]
Oct 30 20:15:56.610440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610423    4519 cpu_manager.go:414] "containerMap: removing container" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:56.610519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610442    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cdcd80-a577-4bdb-8534-a60196f790e9" containerName="cleanup"
Oct 30 20:15:56.610519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610461    4519 cpu_manager.go:414] "containerMap: removing container" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:56.610519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610487    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:56.610519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610499    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:56.610519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610503    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2a5dc7-e640-4fae-abec-0fe6f439634b" containerName="cleanup"
Oct 30 20:15:56.610519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610512    4519 cpu_manager.go:414] "containerMap: removing container" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy"
Oct 30 20:15:56.610519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610517    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610529    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610534    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610567    4519 cpu_manager.go:414] "containerMap: removing container" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610572    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610581    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610585    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610699    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610712    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610719    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62cb1f4-31fd-4e48-9353-ec664afa53bd" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610724    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a475c5c9-8cc0-4243-85b8-df9df8ba14b6" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610730    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9964e85a-1235-4ea4-9e0c-fcfa7339aea2" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610737    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65d56c5-518a-4275-b51f-4dd153ee64d3" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610743    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="14be7e43-88ac-4727-9c8d-cee1e4497ec4" containerName="cleanup"
Oct 30 20:15:56.610746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610750    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6af51b-6caa-4f15-a151-dfc7928ddec8" containerName="cleanup"
Oct 30 20:15:56.611175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610760    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="944c5b58-189c-401a-aeea-2d1f58baeb46" containerName="cleanup"
Oct 30 20:15:56.611175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610766    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f3a318-c311-494c-8152-310d7c3b35e6" containerName="cleanup"
Oct 30 20:15:56.611175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.610772    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb0f2b8-6560-4939-ba71-d1dc2b391a91" containerName="cleanup"
Oct 30 20:15:56.615516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.615461    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.633452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.633427    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"]
Oct 30 20:15:56.640362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.640325    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:56.640506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.640426    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:15:56.706601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.706561    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.706768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.706674    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bzn\" (UniqueName: \"kubernetes.io/projected/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-kube-api-access-v2bzn\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.706956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.706937    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.808908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.808867    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.809116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.808929    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bzn\" (UniqueName: \"kubernetes.io/projected/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-kube-api-access-v2bzn\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.809189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.809151    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.809966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.809942    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.811278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.811255    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.828085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.828058    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bzn\" (UniqueName: \"kubernetes.io/projected/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-kube-api-access-v2bzn\") pod \"maas-api-revoke-sub-b37048f8-f4xxl\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:56.929352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:56.929315    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:15:57.241569 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:15:57.241535    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c715fa8_6f9b_4deb_a6ae_8769ae6ed7bb.slice/crio-33b2a3177698dae5e6cceefaf7291d6443868172bfe3c61a0be4b66e5c2c3a6e WatchSource:0}: Error finding container 33b2a3177698dae5e6cceefaf7291d6443868172bfe3c61a0be4b66e5c2c3a6e: Status 404 returned error can't find the container with id 33b2a3177698dae5e6cceefaf7291d6443868172bfe3c61a0be4b66e5c2c3a6e
Oct 30 20:15:57.243757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:57.243736    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"]
Oct 30 20:15:57.361559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:57.361520    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl" event={"ID":"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb","Type":"ContainerStarted","Data":"33b2a3177698dae5e6cceefaf7291d6443868172bfe3c61a0be4b66e5c2c3a6e"}
Oct 30 20:15:57.551845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:57.551763    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"]
Oct 30 20:15:57.551995 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:57.551975    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy" containerID="cri-o://1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd" gracePeriod=30
Oct 30 20:15:58.365704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.365669    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"]
Oct 30 20:15:58.414501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.384004    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy" containerID="cri-o://77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5" gracePeriod=30
Oct 30 20:15:58.498308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.462562    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"]
Oct 30 20:15:58.532282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.528410    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy" containerID="cri-o://ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf" gracePeriod=30
Oct 30 20:15:58.550873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.550817    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl" event={"ID":"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb","Type":"ContainerStarted","Data":"b2478f27ab21f6eaa6e08ec6d3452bdeff670c2b0e86d335b7c7dd7afe67a7d5"}
Oct 30 20:15:58.695916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.695883    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"]
Oct 30 20:15:58.698032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.697981    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl" podStartSLOduration=2.6979675 podStartE2EDuration="2.6979675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:58.695346409 +0000 UTC m=+2483.196050133" watchObservedRunningTime="2026-10-30 20:15:58.6979675 +0000 UTC m=+2483.198671223"
Oct 30 20:15:58.704255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:58.704230    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4a31d3-rx7fd"]
Oct 30 20:15:59.564155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:59.564117    4519 generic.go:356] "Generic (PLEG): container finished" podID="0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" containerID="b2478f27ab21f6eaa6e08ec6d3452bdeff670c2b0e86d335b7c7dd7afe67a7d5" exitCode=0
Oct 30 20:15:59.564765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:59.564527    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl" event={"ID":"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb","Type":"ContainerDied","Data":"b2478f27ab21f6eaa6e08ec6d3452bdeff670c2b0e86d335b7c7dd7afe67a7d5"}
Oct 30 20:15:59.748001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:15:59.747958    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8a3875-7095-4e8a-bebe-b124eb8f0567" path="/var/lib/kubelet/pods/9c8a3875-7095-4e8a-bebe-b124eb8f0567/volumes"
Oct 30 20:16:00.104913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.104877    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:16:00.124902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.124846    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:16:00.586963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.586935    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:16:00.900291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.900264    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:16:00.917196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.917164    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-kube-api-access-v2bzn\" (UniqueName: \"kubernetes.io/projected/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-kube-api-access-v2bzn\") pod \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") "
Oct 30 20:16:00.917374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.917239    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-xks-ca\") pod \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") "
Oct 30 20:16:00.917435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.917366    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-openshift-ca\") pod \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\" (UID: \"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb\") "
Oct 30 20:16:00.917786 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.917760    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-openshift-ca" pod "0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" (UID: "0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:00.918460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.918436    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:00.919667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.919636    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-kube-api-access-v2bzn" pod "0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" (UID: "0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb"). InnerVolumeSpecName "kube-api-access-v2bzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:00.919807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:00.919788    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-xks-ca" pod "0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" (UID: "0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:01.019688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:01.019642    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bzn\" (UniqueName: \"kubernetes.io/projected/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-kube-api-access-v2bzn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:01.019688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:01.019688    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:01.583327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:01.583294    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"
Oct 30 20:16:01.583553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:01.583328    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl" event={"ID":"0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb","Type":"ContainerDied","Data":"33b2a3177698dae5e6cceefaf7291d6443868172bfe3c61a0be4b66e5c2c3a6e"}
Oct 30 20:16:01.583553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:01.583395    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b2a3177698dae5e6cceefaf7291d6443868172bfe3c61a0be4b66e5c2c3a6e"
Oct 30 20:16:01.733128 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:01.733084    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"]
Oct 30 20:16:01.733708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:01.733430    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" containerID="cri-o://777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f" gracePeriod=30
Oct 30 20:16:02.222879 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:16:02.222616    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:16:02.854137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.854106    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 20:16:02.967037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.966991    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-socket\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967059    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-token\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-token\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967108    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-podinfo\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967170    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-kube-api-access-cfjsw\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-kube-api-access-cfjsw\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967226    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-envoy\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967291    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-credential-socket\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967432    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-data\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967579    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668957ef-930c-4e75-a6f2-3ee872c21db4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668957ef-930c-4e75-a6f2-3ee872c21db4-istiod-ca-cert\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.967835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.967727    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-certs\") pod \"668957ef-930c-4e75-a6f2-3ee872c21db4\" (UID: \"668957ef-930c-4e75-a6f2-3ee872c21db4\") "
Oct 30 20:16:02.968710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.968676    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-certs" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:02.968955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.968931    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-data" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:02.969104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.968945    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-credential-socket" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:02.969383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.969272    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668957ef-930c-4e75-a6f2-3ee872c21db4-istiod-ca-cert" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:02.973010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.971515    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-kube-api-access-cfjsw" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "kube-api-access-cfjsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:02.973121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.973078    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-socket" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:02.976525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.975845    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-podinfo" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:02.980864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.980837    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-token" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:02.980995 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:02.980967    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-envoy" pod "668957ef-930c-4e75-a6f2-3ee872c21db4" (UID: "668957ef-930c-4e75-a6f2-3ee872c21db4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.070133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070086    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070125    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070139    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070152    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfjsw\" (UniqueName: \"kubernetes.io/projected/668957ef-930c-4e75-a6f2-3ee872c21db4-kube-api-access-cfjsw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070167    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070180    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070191    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070205    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668957ef-930c-4e75-a6f2-3ee872c21db4-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.070454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.070217    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668957ef-930c-4e75-a6f2-3ee872c21db4-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.607841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.607793    4519 generic.go:356] "Generic (PLEG): container finished" podID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerID="1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd" exitCode=0
Oct 30 20:16:03.608037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.607961    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"
Oct 30 20:16:03.626754 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.626678    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" event={"ID":"668957ef-930c-4e75-a6f2-3ee872c21db4","Type":"ContainerDied","Data":"1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd"}
Oct 30 20:16:03.626890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.626775    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl" event={"ID":"668957ef-930c-4e75-a6f2-3ee872c21db4","Type":"ContainerDied","Data":"c3d4d2f2157fb76ee88505a4ca1fd3d90f914044ff2af241fcc801f3bc659ecb"}
Oct 30 20:16:03.626890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.626801    4519 scope.go:122] "RemoveContainer" containerID="1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd"
Oct 30 20:16:03.641839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.641808    4519 scope.go:122] "RemoveContainer" containerID="1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd"
Oct 30 20:16:03.642429 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:16:03.642322    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd\": container with ID starting with 1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd not found: ID does not exist" containerID="1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd"
Oct 30 20:16:03.642429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.642364    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd"} err="failed to get container status \"1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd\": rpc error: code = NotFound desc = could not find container \"1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd\": container with ID starting with 1564d9627857fe22e31bb18a2ea80b0be5e66992160114ee03202df5523ac6bd not found: ID does not exist"
Oct 30 20:16:03.690907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.690876    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"]
Oct 30 20:16:03.696924 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.696895    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-kzgzl"]
Oct 30 20:16:03.861258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.860916    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:16:03.915602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.915576    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:16:03.987563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.987534    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-token\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.987741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.987575    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-podinfo\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.988008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.987980    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb1f85f-6380-4563-88e9-b55d8cb73967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb1f85f-6380-4563-88e9-b55d8cb73967-istiod-ca-cert\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.988154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.988067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-podinfo\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.988372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.988345    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb1f85f-6380-4563-88e9-b55d8cb73967-istiod-ca-cert" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:03.988541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.988508    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-envoy\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.988642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.988560    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istiod-ca-cert\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.988642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.988603    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-envoy\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989030    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-token\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-token\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-credential-socket\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989146    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-data\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989215    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-certs\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989282    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-socket\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989312    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-credential-socket\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989348    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-kube-api-access-6sfxg\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-kube-api-access-6sfxg\") pod \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\" (UID: \"ba5039f7-483c-4f5f-b5fb-79cacbe254fb\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989407    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-data\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989534    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-socket\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989605    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-kube-api-access-66sxm\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-kube-api-access-66sxm\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.989650    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-certs\") pod \"6bb1f85f-6380-4563-88e9-b55d8cb73967\" (UID: \"6bb1f85f-6380-4563-88e9-b55d8cb73967\") "
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.990392    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb1f85f-6380-4563-88e9-b55d8cb73967-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.990616    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-credential-socket" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.990627    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istiod-ca-cert" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.990762    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-podinfo" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.990860    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-credential-socket" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.990951    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-certs" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.990963    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-socket" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.991057    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-envoy" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.991137    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-data" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.991748    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-data" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.991813    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-socket" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.991949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.991925    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-certs" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.993410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.993383    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-token" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:03.993590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.993517    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-kube-api-access-6sfxg" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "kube-api-access-6sfxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:03.993695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.993633    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-podinfo" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:03.993888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.993869    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-kube-api-access-66sxm" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "kube-api-access-66sxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:03.995130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.995098    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-envoy" pod "ba5039f7-483c-4f5f-b5fb-79cacbe254fb" (UID: "ba5039f7-483c-4f5f-b5fb-79cacbe254fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.995326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:03.995260    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-token" pod "6bb1f85f-6380-4563-88e9-b55d8cb73967" (UID: "6bb1f85f-6380-4563-88e9-b55d8cb73967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:04.092120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092030    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092067    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092086    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfxg\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-kube-api-access-6sfxg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092101    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092115    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092129    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66sxm\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-kube-api-access-66sxm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092143    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092157    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092170    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092183    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092195    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092209    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092221    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092234    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb1f85f-6380-4563-88e9-b55d8cb73967-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092246    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092260    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.092431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.092272    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5039f7-483c-4f5f-b5fb-79cacbe254fb-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:04.481401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.481348    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:16:04.481671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.481428    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:16:04.618305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.618265    4519 generic.go:356] "Generic (PLEG): container finished" podID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerID="77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5" exitCode=0
Oct 30 20:16:04.618386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.618317    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"
Oct 30 20:16:04.618386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.618359    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" event={"ID":"6bb1f85f-6380-4563-88e9-b55d8cb73967","Type":"ContainerDied","Data":"77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5"}
Oct 30 20:16:04.618532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.618425    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl" event={"ID":"6bb1f85f-6380-4563-88e9-b55d8cb73967","Type":"ContainerDied","Data":"2f540d8c6968eacf7f77029ab3b1181ec155f379d68ba50f2781c78acdf05d4e"}
Oct 30 20:16:04.618532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.618460    4519 scope.go:122] "RemoveContainer" containerID="77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5"
Oct 30 20:16:04.621891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.621860    4519 generic.go:356] "Generic (PLEG): container finished" podID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerID="ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf" exitCode=0
Oct 30 20:16:04.622020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.621906    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" event={"ID":"ba5039f7-483c-4f5f-b5fb-79cacbe254fb","Type":"ContainerDied","Data":"ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf"}
Oct 30 20:16:04.622020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.621965    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj" event={"ID":"ba5039f7-483c-4f5f-b5fb-79cacbe254fb","Type":"ContainerDied","Data":"0afeb32d8553e98aee0c67665e91ff94280f42eeda72122f02a5e1e7378de182"}
Oct 30 20:16:04.622207 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.622181    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"
Oct 30 20:16:04.633510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.633488    4519 scope.go:122] "RemoveContainer" containerID="8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa"
Oct 30 20:16:04.643851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.643831    4519 scope.go:122] "RemoveContainer" containerID="77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5"
Oct 30 20:16:04.644155 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:16:04.644130    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5\": container with ID starting with 77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5 not found: ID does not exist" containerID="77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5"
Oct 30 20:16:04.644230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.644161    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5"} err="failed to get container status \"77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5\": rpc error: code = NotFound desc = could not find container \"77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5\": container with ID starting with 77830120fcf13ca239e59914e4233eb3bf48061258c37e5d601f4b39f2143cd5 not found: ID does not exist"
Oct 30 20:16:04.644230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.644179    4519 scope.go:122] "RemoveContainer" containerID="8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa"
Oct 30 20:16:04.644495 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:16:04.644443    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa\": container with ID starting with 8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa not found: ID does not exist" containerID="8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa"
Oct 30 20:16:04.644568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.644502    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa"} err="failed to get container status \"8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa\": rpc error: code = NotFound desc = could not find container \"8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa\": container with ID starting with 8b12d93a1ff1431dab53a31d738d9ef2fad09feadd141924b70bfcbe80c282aa not found: ID does not exist"
Oct 30 20:16:04.644568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.644523    4519 scope.go:122] "RemoveContainer" containerID="ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf"
Oct 30 20:16:04.654629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.654609    4519 scope.go:122] "RemoveContainer" containerID="ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf"
Oct 30 20:16:04.654906 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:16:04.654887    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf\": container with ID starting with ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf not found: ID does not exist" containerID="ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf"
Oct 30 20:16:04.655017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.654916    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf"} err="failed to get container status \"ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf\": rpc error: code = NotFound desc = could not find container \"ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf\": container with ID starting with ce41837291579ba273c762e19cb467a35f42605136991e212471844568e878bf not found: ID does not exist"
Oct 30 20:16:04.672169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.672125    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"]
Oct 30 20:16:04.683522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.683495    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-dd9wl"]
Oct 30 20:16:04.719870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.719839    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"]
Oct 30 20:16:04.727079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:04.727038    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-jzjfj"]
Oct 30 20:16:05.616527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:05.616487    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" path="/var/lib/kubelet/pods/668957ef-930c-4e75-a6f2-3ee872c21db4/volumes"
Oct 30 20:16:05.617087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:05.617070    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" path="/var/lib/kubelet/pods/6bb1f85f-6380-4563-88e9-b55d8cb73967/volumes"
Oct 30 20:16:05.617584 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:05.617570    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" path="/var/lib/kubelet/pods/ba5039f7-483c-4f5f-b5fb-79cacbe254fb/volumes"
Oct 30 20:16:07.142199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.142170    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 20:16:07.241746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.241706    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-socket\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.241960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.241815    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d6ba7b-880c-4d52-946a-30d883491b8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6ba7b-880c-4d52-946a-30d883491b8b-istiod-ca-cert\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.241960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.241846    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-credential-socket\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.242101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.241961    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-kube-api-access-ntm9c\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-kube-api-access-ntm9c\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.242101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242016    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-socket" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.242101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242029    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-token\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-token\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.242277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242158    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-certs\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.242334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242245    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-envoy\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.242397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242335    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-data\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.242397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242331    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-credential-socket" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.242577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242413    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-podinfo\") pod \"d9d6ba7b-880c-4d52-946a-30d883491b8b\" (UID: \"d9d6ba7b-880c-4d52-946a-30d883491b8b\") "
Oct 30 20:16:07.242577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242560    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-certs" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.242791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.242758    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-data" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.243037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.243018    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.243105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.243043    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.243105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.243057    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.243105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.243073    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.243715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.243690    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d6ba7b-880c-4d52-946a-30d883491b8b-istiod-ca-cert" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:07.245165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.245137    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-envoy" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:07.245281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.245261    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-kube-api-access-ntm9c" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "kube-api-access-ntm9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:07.245646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.245623    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-podinfo" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:07.245877 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.245858    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-token" pod "d9d6ba7b-880c-4d52-946a-30d883491b8b" (UID: "d9d6ba7b-880c-4d52-946a-30d883491b8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:07.344986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.344883    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntm9c\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-kube-api-access-ntm9c\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.344986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.344929    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.344986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.344944    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.344986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.344953    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d6ba7b-880c-4d52-946a-30d883491b8b-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.344986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.344964    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6ba7b-880c-4d52-946a-30d883491b8b-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:07.669248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.669213    4519 generic.go:356] "Generic (PLEG): container finished" podID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerID="777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f" exitCode=0
Oct 30 20:16:07.669489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.669273    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"
Oct 30 20:16:07.669489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.669299    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" event={"ID":"d9d6ba7b-880c-4d52-946a-30d883491b8b","Type":"ContainerDied","Data":"777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f"}
Oct 30 20:16:07.669489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.669367    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm" event={"ID":"d9d6ba7b-880c-4d52-946a-30d883491b8b","Type":"ContainerDied","Data":"dc71a85ec93a4a27b55e70882ed2cfaf8d6e3bdb393d560f884e9fc70b754bae"}
Oct 30 20:16:07.669489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.669396    4519 scope.go:122] "RemoveContainer" containerID="777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f"
Oct 30 20:16:07.684070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.684053    4519 scope.go:122] "RemoveContainer" containerID="777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f"
Oct 30 20:16:07.684372 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:16:07.684349    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f\": container with ID starting with 777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f not found: ID does not exist" containerID="777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f"
Oct 30 20:16:07.684498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.684377    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f"} err="failed to get container status \"777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f\": rpc error: code = NotFound desc = could not find container \"777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f\": container with ID starting with 777a6f8b1f850c7d50885f93392d1c301f9d2c25947325d8a3448f97f2d64b6f not found: ID does not exist"
Oct 30 20:16:07.728633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.728607    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"]
Oct 30 20:16:07.740079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.740053    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-nw6nm"]
Oct 30 20:16:07.932072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932034    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"]
Oct 30 20:16:07.932804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932786    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932807    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932844    4519 cpu_manager.go:414] "containerMap: removing container" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932852    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db80b810-4764-4663-839e-1bb464bafcf8" containerName="istio-proxy"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932862    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" containerName="revoke-keys"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932870    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" containerName="revoke-keys"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932891    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932899    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932913    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy"
Oct 30 20:16:07.932916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932920    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932931    4519 cpu_manager.go:414] "containerMap: removing container" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932939    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932973    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.932980    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.933089    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.933104    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.933117    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" containerName="revoke-keys"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.933128    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5039f7-483c-4f5f-b5fb-79cacbe254fb" containerName="istio-proxy"
Oct 30 20:16:07.933438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.933139    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="668957ef-930c-4e75-a6f2-3ee872c21db4" containerName="istio-proxy"
Oct 30 20:16:07.937293 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.937269    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.953918    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-credential-socket\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954080    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-envoy\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954216    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-podinfo\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954286    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-token\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954513    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-data\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954584    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-certs\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istiod-ca-cert\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954660    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvn4\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-kube-api-access-npvn4\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.954710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.954688    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-socket\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:07.957458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.956867    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5314c2-openshift-default-dockercfg-pqmzn\""
Oct 30 20:16:07.961728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:07.961702    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"]
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087495    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-token\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087582    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-data\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087624    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-certs\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087681    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istiod-ca-cert\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087711    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvn4\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-kube-api-access-npvn4\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087742    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-socket\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087855    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-credential-socket\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.087913    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-envoy\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.089525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.088048    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-podinfo\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.118744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.091262    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-data\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.118744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.091881    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-socket\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.118744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.092353    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-credential-socket\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.118744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.092610    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-certs\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.122296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.122249    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istiod-ca-cert\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.141326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.133784    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-envoy\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.152180 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.147957    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-token\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.152180 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.148514    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-podinfo\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.152180 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.149109    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvn4\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-kube-api-access-npvn4\") pod \"e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") " pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.297603 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.288279    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:08.685179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.685143    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"]
Oct 30 20:16:08.685998 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:16:08.685967    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c9ead9_68b8_41cc_8ef8_6f4cdb63cee3.slice/crio-1ef627c33e49822d7255f122a3b65db3b94e08ded6f1bf9780ddba27f22046b7 WatchSource:0}: Error finding container 1ef627c33e49822d7255f122a3b65db3b94e08ded6f1bf9780ddba27f22046b7: Status 404 returned error can't find the container with id 1ef627c33e49822d7255f122a3b65db3b94e08ded6f1bf9780ddba27f22046b7
Oct 30 20:16:08.688867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.688828    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:16:08.688974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.688901    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:16:08.688974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:08.688932    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:16:09.037890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:09.037803    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"]
Oct 30 20:16:09.049679 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:09.049582    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e136ff9b-rkslw"]
Oct 30 20:16:09.614211 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:09.614179    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0df3be-dcb7-4f1b-b15a-c1fd00de5096" path="/var/lib/kubelet/pods/5d0df3be-dcb7-4f1b-b15a-c1fd00de5096/volumes"
Oct 30 20:16:09.614739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:09.614723    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d6ba7b-880c-4d52-946a-30d883491b8b" path="/var/lib/kubelet/pods/d9d6ba7b-880c-4d52-946a-30d883491b8b/volumes"
Oct 30 20:16:09.695267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:09.695226    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" event={"ID":"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3","Type":"ContainerStarted","Data":"604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef"}
Oct 30 20:16:09.695591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:09.695291    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" event={"ID":"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3","Type":"ContainerStarted","Data":"1ef627c33e49822d7255f122a3b65db3b94e08ded6f1bf9780ddba27f22046b7"}
Oct 30 20:16:09.744352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:09.744297    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podStartSLOduration=2.7442827640000003 podStartE2EDuration="2.744282764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:09.740245162 +0000 UTC m=+2494.240948878" watchObservedRunningTime="2026-10-30 20:16:09.744282764 +0000 UTC m=+2494.244986488"
Oct 30 20:16:10.289432 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:10.289390    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:10.297845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:10.297813    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:10.708285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:10.708256    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:10.709343 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:10.709321    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:16:21.029699 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:21.029664    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"]
Oct 30 20:16:21.041341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:21.041315    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee823fe0-sdbdv"]
Oct 30 20:16:21.614035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:21.613996    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eaf708b-2ec3-45e3-bd61-164cbb491ac8" path="/var/lib/kubelet/pods/4eaf708b-2ec3-45e3-bd61-164cbb491ac8/volumes"
Oct 30 20:16:26.703657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.703605    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"]
Oct 30 20:16:26.704661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.704629    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb1f85f-6380-4563-88e9-b55d8cb73967" containerName="istio-proxy"
Oct 30 20:16:26.731488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.730743    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.742256 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.742219    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"]
Oct 30 20:16:26.748852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.748815    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:26.762751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.762726    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:16:26.825155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.825119    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.825328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.825164    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9m4\" (UniqueName: \"kubernetes.io/projected/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-kube-api-access-wg9m4\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.825431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.825407    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.926691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.926651    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.926885 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.926698    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9m4\" (UniqueName: \"kubernetes.io/projected/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-kube-api-access-wg9m4\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.926957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.926924    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.927514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.927456    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.929371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.929343    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:26.945726 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:26.945701    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9m4\" (UniqueName: \"kubernetes.io/projected/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-kube-api-access-wg9m4\") pod \"maas-api-revoke-sub-79cb8475-2djbx\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:27.065208 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:27.065131    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:27.380350 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:16:27.380304    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc5b6a6_e216_41dd_8fd1_3fb35a12aae1.slice/crio-1b5debd0d180fd8ee66686b912fc028f08e611d60361661e368b39ced3677c67 WatchSource:0}: Error finding container 1b5debd0d180fd8ee66686b912fc028f08e611d60361661e368b39ced3677c67: Status 404 returned error can't find the container with id 1b5debd0d180fd8ee66686b912fc028f08e611d60361661e368b39ced3677c67
Oct 30 20:16:27.382840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:27.382810    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"]
Oct 30 20:16:27.923817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:27.923775    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx" event={"ID":"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1","Type":"ContainerStarted","Data":"17e797cd2f1eec221fa4b7cabb1669cf4686cd9b485791fb490e58b24d1fe357"}
Oct 30 20:16:27.924332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:27.923839    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx" event={"ID":"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1","Type":"ContainerStarted","Data":"1b5debd0d180fd8ee66686b912fc028f08e611d60361661e368b39ced3677c67"}
Oct 30 20:16:27.965695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:27.965632    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx" podStartSLOduration=1.96561873 podStartE2EDuration="1.96561873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:27.961715236 +0000 UTC m=+2512.462418963" watchObservedRunningTime="2026-10-30 20:16:27.96561873 +0000 UTC m=+2512.466322453"
Oct 30 20:16:28.933556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:28.933520    4519 generic.go:356] "Generic (PLEG): container finished" podID="fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" containerID="17e797cd2f1eec221fa4b7cabb1669cf4686cd9b485791fb490e58b24d1fe357" exitCode=0
Oct 30 20:16:28.933960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:28.933610    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx" event={"ID":"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1","Type":"ContainerDied","Data":"17e797cd2f1eec221fa4b7cabb1669cf4686cd9b485791fb490e58b24d1fe357"}
Oct 30 20:16:29.078802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.078763    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"]
Oct 30 20:16:29.082859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.082840    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.101249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.101218    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"]
Oct 30 20:16:29.160791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.160752    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.160950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.160884    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlrt\" (UniqueName: \"kubernetes.io/projected/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-kube-api-access-njlrt\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.161036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.160964    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.262121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.262030    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.262295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.262120    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlrt\" (UniqueName: \"kubernetes.io/projected/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-kube-api-access-njlrt\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.262295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.262159    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.262883 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.262850    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.264438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.264410    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.280958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.280937    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlrt\" (UniqueName: \"kubernetes.io/projected/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-kube-api-access-njlrt\") pod \"maas-api-revoke-sub-59e7611d-4kjfr\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.397509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.397477    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:29.701388 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:16:29.701354    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a6cd71_e9f4_4cde_b7de_5e0bf4e867c9.slice/crio-cb2c18c64bc16589dc423c38dbdfa106ff790da3124bbec2f6e65a052eee6445 WatchSource:0}: Error finding container cb2c18c64bc16589dc423c38dbdfa106ff790da3124bbec2f6e65a052eee6445: Status 404 returned error can't find the container with id cb2c18c64bc16589dc423c38dbdfa106ff790da3124bbec2f6e65a052eee6445
Oct 30 20:16:29.702560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.702534    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"]
Oct 30 20:16:29.942611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:29.942578    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr" event={"ID":"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9","Type":"ContainerStarted","Data":"cb2c18c64bc16589dc423c38dbdfa106ff790da3124bbec2f6e65a052eee6445"}
Oct 30 20:16:30.101221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.101190    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:30.185801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.185770    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-xks-ca\") pod \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") "
Oct 30 20:16:30.185952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.185937    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-openshift-ca\") pod \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") "
Oct 30 20:16:30.186104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.186086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-kube-api-access-wg9m4\" (UniqueName: \"kubernetes.io/projected/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-kube-api-access-wg9m4\") pod \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\" (UID: \"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1\") "
Oct 30 20:16:30.186374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.186336    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-openshift-ca" pod "fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" (UID: "fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:30.186731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.186707    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:30.187970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.187946    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-xks-ca" pod "fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" (UID: "fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:30.188129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.188112    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-kube-api-access-wg9m4" pod "fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" (UID: "fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1"). InnerVolumeSpecName "kube-api-access-wg9m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:30.288690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.288651    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:30.288837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.288692    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9m4\" (UniqueName: \"kubernetes.io/projected/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1-kube-api-access-wg9m4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:30.952620 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.952585    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"
Oct 30 20:16:30.953044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.952608    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx" event={"ID":"fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1","Type":"ContainerDied","Data":"1b5debd0d180fd8ee66686b912fc028f08e611d60361661e368b39ced3677c67"}
Oct 30 20:16:30.953044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.952700    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b5debd0d180fd8ee66686b912fc028f08e611d60361661e368b39ced3677c67"
Oct 30 20:16:30.954707 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.954675    4519 generic.go:356] "Generic (PLEG): container finished" podID="72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" containerID="c521ae4d4ccdf24d0a310c188a4ae05ba69b08257d58f74867e96c204a42378d" exitCode=0
Oct 30 20:16:30.954860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:30.954763    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr" event={"ID":"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9","Type":"ContainerDied","Data":"c521ae4d4ccdf24d0a310c188a4ae05ba69b08257d58f74867e96c204a42378d"}
Oct 30 20:16:32.169514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.168424    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:32.230929 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:16:32.230839    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:16:32.328709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.328675    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-kube-api-access-njlrt\" (UniqueName: \"kubernetes.io/projected/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-kube-api-access-njlrt\") pod \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") "
Oct 30 20:16:32.328893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.328739    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-xks-ca\") pod \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") "
Oct 30 20:16:32.328893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.328835    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-openshift-ca\") pod \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\" (UID: \"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9\") "
Oct 30 20:16:32.329245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.329222    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-openshift-ca" pod "72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" (UID: "72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:32.330983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.330953    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-kube-api-access-njlrt" pod "72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" (UID: "72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9"). InnerVolumeSpecName "kube-api-access-njlrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:32.330983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.330968    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-xks-ca" pod "72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" (UID: "72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:32.431082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.431049    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:32.431082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.431078    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:32.431082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.431089    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njlrt\" (UniqueName: \"kubernetes.io/projected/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9-kube-api-access-njlrt\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:32.977345 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.977310    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr" event={"ID":"72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9","Type":"ContainerDied","Data":"cb2c18c64bc16589dc423c38dbdfa106ff790da3124bbec2f6e65a052eee6445"}
Oct 30 20:16:32.977632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.977355    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"
Oct 30 20:16:32.977632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:32.977363    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2c18c64bc16589dc423c38dbdfa106ff790da3124bbec2f6e65a052eee6445"
Oct 30 20:16:39.495452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.495416    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"]
Oct 30 20:16:39.495999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.495983    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" containerName="revoke-keys"
Oct 30 20:16:39.496046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.496000    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" containerName="revoke-keys"
Oct 30 20:16:39.496046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.496022    4519 cpu_manager.go:414] "containerMap: removing container" podUID="72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" containerName="revoke-keys"
Oct 30 20:16:39.496046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.496027    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" containerName="revoke-keys"
Oct 30 20:16:39.496147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.496126    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" containerName="revoke-keys"
Oct 30 20:16:39.496147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.496138    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" containerName="revoke-keys"
Oct 30 20:16:39.499680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.499654    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.518217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.518188    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"]
Oct 30 20:16:39.525426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.525391    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:39.525594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.525437    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:16:39.635194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.635155    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.635404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.635379    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.635549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.635532    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcpt\" (UniqueName: \"kubernetes.io/projected/42f94e19-8134-4cf0-8815-462f1a84034a-kube-api-access-8kcpt\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.736805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.736772    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.736991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.736856    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcpt\" (UniqueName: \"kubernetes.io/projected/42f94e19-8134-4cf0-8815-462f1a84034a-kube-api-access-8kcpt\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.736991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.736920    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.737808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.737783    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.739240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.739217    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.756032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.755963    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcpt\" (UniqueName: \"kubernetes.io/projected/42f94e19-8134-4cf0-8815-462f1a84034a-kube-api-access-8kcpt\") pod \"maas-api-revoke-sub-94268f94-99595\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:39.813663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:39.813634    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:40.126645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:40.126608    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"]
Oct 30 20:16:41.100862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:41.100820    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595" event={"ID":"42f94e19-8134-4cf0-8815-462f1a84034a","Type":"ContainerStarted","Data":"6d76cb83ddda7443df947aba12885c257e5a6ceccb55ba6101c1793c147ce4dd"}
Oct 30 20:16:41.102663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:41.102634    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595" event={"ID":"42f94e19-8134-4cf0-8815-462f1a84034a","Type":"ContainerStarted","Data":"e7e27a7a8434338e50e5064c347273c49213294906c54df960e30c47f560c474"}
Oct 30 20:16:41.141366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:41.141306    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595" podStartSLOduration=2.141287968 podStartE2EDuration="2.141287968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:41.139920294 +0000 UTC m=+2525.640624019" watchObservedRunningTime="2026-10-30 20:16:41.141287968 +0000 UTC m=+2525.641991692"
Oct 30 20:16:42.110125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:42.110082    4519 generic.go:356] "Generic (PLEG): container finished" podID="42f94e19-8134-4cf0-8815-462f1a84034a" containerID="6d76cb83ddda7443df947aba12885c257e5a6ceccb55ba6101c1793c147ce4dd" exitCode=0
Oct 30 20:16:42.110593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:42.110171    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595" event={"ID":"42f94e19-8134-4cf0-8815-462f1a84034a","Type":"ContainerDied","Data":"6d76cb83ddda7443df947aba12885c257e5a6ceccb55ba6101c1793c147ce4dd"}
Oct 30 20:16:43.280825 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.280796    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:43.404916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.404827    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-xks-ca\") pod \"42f94e19-8134-4cf0-8815-462f1a84034a\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") "
Oct 30 20:16:43.404916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.404877    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-openshift-ca\") pod \"42f94e19-8134-4cf0-8815-462f1a84034a\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") "
Oct 30 20:16:43.405161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.405098    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f94e19-8134-4cf0-8815-462f1a84034a-kube-api-access-8kcpt\" (UniqueName: \"kubernetes.io/projected/42f94e19-8134-4cf0-8815-462f1a84034a-kube-api-access-8kcpt\") pod \"42f94e19-8134-4cf0-8815-462f1a84034a\" (UID: \"42f94e19-8134-4cf0-8815-462f1a84034a\") "
Oct 30 20:16:43.405349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.405326    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-openshift-ca" pod "42f94e19-8134-4cf0-8815-462f1a84034a" (UID: "42f94e19-8134-4cf0-8815-462f1a84034a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:43.405778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.405747    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:43.407195 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.407163    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-xks-ca" pod "42f94e19-8134-4cf0-8815-462f1a84034a" (UID: "42f94e19-8134-4cf0-8815-462f1a84034a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:43.407307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.407284    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f94e19-8134-4cf0-8815-462f1a84034a-kube-api-access-8kcpt" pod "42f94e19-8134-4cf0-8815-462f1a84034a" (UID: "42f94e19-8134-4cf0-8815-462f1a84034a"). InnerVolumeSpecName "kube-api-access-8kcpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:43.506795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.506759    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcpt\" (UniqueName: \"kubernetes.io/projected/42f94e19-8134-4cf0-8815-462f1a84034a-kube-api-access-8kcpt\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:43.506795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:43.506789    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f94e19-8134-4cf0-8815-462f1a84034a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:16:44.132732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:44.132685    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595" event={"ID":"42f94e19-8134-4cf0-8815-462f1a84034a","Type":"ContainerDied","Data":"e7e27a7a8434338e50e5064c347273c49213294906c54df960e30c47f560c474"}
Oct 30 20:16:44.132901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:44.132748    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"
Oct 30 20:16:44.132901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:44.132751    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e27a7a8434338e50e5064c347273c49213294906c54df960e30c47f560c474"
Oct 30 20:16:58.469514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.467209    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"]
Oct 30 20:16:58.469514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.467797    4519 cpu_manager.go:414] "containerMap: removing container" podUID="42f94e19-8134-4cf0-8815-462f1a84034a" containerName="revoke-keys"
Oct 30 20:16:58.469514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.467810    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f94e19-8134-4cf0-8815-462f1a84034a" containerName="revoke-keys"
Oct 30 20:16:58.469514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.467924    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f94e19-8134-4cf0-8815-462f1a84034a" containerName="revoke-keys"
Oct 30 20:16:58.473653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.473629    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.494344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.494220    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"]
Oct 30 20:16:58.500517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.500461    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"auth-test-model-27a4d3-kserve-self-signed-certs\""
Oct 30 20:16:58.500670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.500575    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"default-dockercfg-2spx4\""
Oct 30 20:16:58.500670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.500574    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"kube-root-ca.crt\""
Oct 30 20:16:58.500670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.500663    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"openshift-service-ca.crt\""
Oct 30 20:16:58.533702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.533670    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-dshm\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.533843 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.533824    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwv2\" (UniqueName: \"kubernetes.io/projected/8dcf626d-9c08-4db6-b200-b0d17f8b504c-kube-api-access-6xwv2\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.533907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.533892    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-home\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.533986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.533966    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-model-cache\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.534092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.534077    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tmp-dir\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.534203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.534177    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tls-certs\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635092    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tmp-dir\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tls-certs\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635268    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-dshm\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635462 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635372    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwv2\" (UniqueName: \"kubernetes.io/projected/8dcf626d-9c08-4db6-b200-b0d17f8b504c-kube-api-access-6xwv2\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635461    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-home\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635599    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tmp-dir\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635760    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-model-cache\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.635781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.635779    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-home\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.636065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.636042    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-model-cache\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.637792 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.637753    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-dshm\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.638397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.638373    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tls-certs\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.657999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.657971    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwv2\" (UniqueName: \"kubernetes.io/projected/8dcf626d-9c08-4db6-b200-b0d17f8b504c-kube-api-access-6xwv2\") pod \"auth-test-model-27a4d3-kserve-67479f569-tt5zp\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:58.798023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:58.797906    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:16:59.152983 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:16:59.152941    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dcf626d_9c08_4db6_b200_b0d17f8b504c.slice/crio-2897445b4b167d0825e961fa4d655eeccfdab487b430887b4829ab4f80090330 WatchSource:0}: Error finding container 2897445b4b167d0825e961fa4d655eeccfdab487b430887b4829ab4f80090330: Status 404 returned error can't find the container with id 2897445b4b167d0825e961fa4d655eeccfdab487b430887b4829ab4f80090330
Oct 30 20:16:59.153271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:59.153246    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"]
Oct 30 20:16:59.264946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:59.264903    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp" event={"ID":"8dcf626d-9c08-4db6-b200-b0d17f8b504c","Type":"ContainerStarted","Data":"1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6"}
Oct 30 20:16:59.264946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:59.264950    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp" event={"ID":"8dcf626d-9c08-4db6-b200-b0d17f8b504c","Type":"ContainerStarted","Data":"2897445b4b167d0825e961fa4d655eeccfdab487b430887b4829ab4f80090330"}
Oct 30 20:16:59.304698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:59.304637    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp" podStartSLOduration=1.3046226029999999 podStartE2EDuration="1.304622603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:59.303252421 +0000 UTC m=+2543.803956147" watchObservedRunningTime="2026-10-30 20:16:59.304622603 +0000 UTC m=+2543.805326371"
Oct 30 20:16:59.806307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:16:59.806266    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:17:01.038845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.038784    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"]
Oct 30 20:17:01.044132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.044105    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.065810 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.065781    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"]
Oct 30 20:17:01.067731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.067707    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:01.068013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.067998    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:17:01.203985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.203935    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/415a3dfe-bd6e-4878-a940-c1b34360ccef-kube-api-access-2hmbc\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.204194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.204066    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.204310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.204289    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.305610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.305518    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/415a3dfe-bd6e-4878-a940-c1b34360ccef-kube-api-access-2hmbc\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.305610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.305597    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.305852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.305792    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.306539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.306512    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.308190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.308160    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.325840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.325812    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/415a3dfe-bd6e-4878-a940-c1b34360ccef-kube-api-access-2hmbc\") pod \"maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.368713 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.368680    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:01.689257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:01.689222    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"]
Oct 30 20:17:02.223078 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:17:02.223045    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:17:02.305133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:02.302362    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh" event={"ID":"415a3dfe-bd6e-4878-a940-c1b34360ccef","Type":"ContainerStarted","Data":"02380a49c2a6ee9e98833ca9c6fa7c8bd66b8b8cded639897c540a4047d3584c"}
Oct 30 20:17:02.305133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:02.302434    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh" event={"ID":"415a3dfe-bd6e-4878-a940-c1b34360ccef","Type":"ContainerStarted","Data":"a4c55fbaf3e41b15a428cff5d21d8252a4c51f24919ba94d51c8c64776b632d0"}
Oct 30 20:17:02.345047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:02.344946    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh" podStartSLOduration=1.3449333399999999 podStartE2EDuration="1.34493334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:02.341203361 +0000 UTC m=+2546.841907085" watchObservedRunningTime="2026-10-30 20:17:02.34493334 +0000 UTC m=+2546.845637064"
Oct 30 20:17:03.315424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:03.315389    4519 generic.go:356] "Generic (PLEG): container finished" podID="415a3dfe-bd6e-4878-a940-c1b34360ccef" containerID="02380a49c2a6ee9e98833ca9c6fa7c8bd66b8b8cded639897c540a4047d3584c" exitCode=0
Oct 30 20:17:03.315924 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:03.315499    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh" event={"ID":"415a3dfe-bd6e-4878-a940-c1b34360ccef","Type":"ContainerDied","Data":"02380a49c2a6ee9e98833ca9c6fa7c8bd66b8b8cded639897c540a4047d3584c"}
Oct 30 20:17:03.606566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:03.604652    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"]
Oct 30 20:17:04.518182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.518153    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:04.666260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.666224    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-openshift-ca\") pod \"415a3dfe-bd6e-4878-a940-c1b34360ccef\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") "
Oct 30 20:17:04.666434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.666361    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415a3dfe-bd6e-4878-a940-c1b34360ccef-kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/415a3dfe-bd6e-4878-a940-c1b34360ccef-kube-api-access-2hmbc\") pod \"415a3dfe-bd6e-4878-a940-c1b34360ccef\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") "
Oct 30 20:17:04.666531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.666430    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-xks-ca\") pod \"415a3dfe-bd6e-4878-a940-c1b34360ccef\" (UID: \"415a3dfe-bd6e-4878-a940-c1b34360ccef\") "
Oct 30 20:17:04.666681 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.666650    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-openshift-ca" pod "415a3dfe-bd6e-4878-a940-c1b34360ccef" (UID: "415a3dfe-bd6e-4878-a940-c1b34360ccef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:04.667066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.667042    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:04.669137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.669112    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-xks-ca" pod "415a3dfe-bd6e-4878-a940-c1b34360ccef" (UID: "415a3dfe-bd6e-4878-a940-c1b34360ccef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:04.669243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.669226    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415a3dfe-bd6e-4878-a940-c1b34360ccef-kube-api-access-2hmbc" pod "415a3dfe-bd6e-4878-a940-c1b34360ccef" (UID: "415a3dfe-bd6e-4878-a940-c1b34360ccef"). InnerVolumeSpecName "kube-api-access-2hmbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:04.768928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.768891    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415a3dfe-bd6e-4878-a940-c1b34360ccef-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:04.769098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:04.768930    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/415a3dfe-bd6e-4878-a940-c1b34360ccef-kube-api-access-2hmbc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:05.336410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:05.336371    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"
Oct 30 20:17:05.336616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:05.336404    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh" event={"ID":"415a3dfe-bd6e-4878-a940-c1b34360ccef","Type":"ContainerDied","Data":"a4c55fbaf3e41b15a428cff5d21d8252a4c51f24919ba94d51c8c64776b632d0"}
Oct 30 20:17:05.336616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:05.336456    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4c55fbaf3e41b15a428cff5d21d8252a4c51f24919ba94d51c8c64776b632d0"
Oct 30 20:17:08.806431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:08.806387    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:17:08.819224 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:08.819196    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:17:09.384491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:09.384444    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:17:18.636672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:18.636621    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x" podUID="75f25887-41c2-498f-87c9-2299fa9008c2" containerName="main" containerID="cri-o://bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622" gracePeriod=45
Oct 30 20:17:18.760542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:18.758266    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x" podUID="75f25887-41c2-498f-87c9-2299fa9008c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.219:8000/ready\": dial tcp 10.133.0.219:8000: connect: connection refused"
Oct 30 20:17:19.001816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.001786    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:17:19.194700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.194654    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-tmp-dir\") pod \"75f25887-41c2-498f-87c9-2299fa9008c2\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") "
Oct 30 20:17:19.196664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.195067    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-tmp-dir" pod "75f25887-41c2-498f-87c9-2299fa9008c2" (UID: "75f25887-41c2-498f-87c9-2299fa9008c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:19.197056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.196958    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f25887-41c2-498f-87c9-2299fa9008c2-kube-api-access-cgtx5\" (UniqueName: \"kubernetes.io/projected/75f25887-41c2-498f-87c9-2299fa9008c2-kube-api-access-cgtx5\") pod \"75f25887-41c2-498f-87c9-2299fa9008c2\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") "
Oct 30 20:17:19.197405 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.197384    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-model-cache\") pod \"75f25887-41c2-498f-87c9-2299fa9008c2\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") "
Oct 30 20:17:19.198512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.197584    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-home\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-home\") pod \"75f25887-41c2-498f-87c9-2299fa9008c2\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") "
Oct 30 20:17:19.198512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.198320    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-dshm\") pod \"75f25887-41c2-498f-87c9-2299fa9008c2\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") "
Oct 30 20:17:19.198512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.198420    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f25887-41c2-498f-87c9-2299fa9008c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/75f25887-41c2-498f-87c9-2299fa9008c2-tls-certs\") pod \"75f25887-41c2-498f-87c9-2299fa9008c2\" (UID: \"75f25887-41c2-498f-87c9-2299fa9008c2\") "
Oct 30 20:17:19.198804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.198166    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-home" pod "75f25887-41c2-498f-87c9-2299fa9008c2" (UID: "75f25887-41c2-498f-87c9-2299fa9008c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:19.199573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.199367    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:19.199724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.198217    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-model-cache" pod "75f25887-41c2-498f-87c9-2299fa9008c2" (UID: "75f25887-41c2-498f-87c9-2299fa9008c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:19.200698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.200675    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f25887-41c2-498f-87c9-2299fa9008c2-kube-api-access-cgtx5" pod "75f25887-41c2-498f-87c9-2299fa9008c2" (UID: "75f25887-41c2-498f-87c9-2299fa9008c2"). InnerVolumeSpecName "kube-api-access-cgtx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:19.201342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.201311    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f25887-41c2-498f-87c9-2299fa9008c2-tls-certs" pod "75f25887-41c2-498f-87c9-2299fa9008c2" (UID: "75f25887-41c2-498f-87c9-2299fa9008c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:19.202062 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.202043    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-dshm" pod "75f25887-41c2-498f-87c9-2299fa9008c2" (UID: "75f25887-41c2-498f-87c9-2299fa9008c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:19.300438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.300357    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:19.300438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.300389    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:19.300438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.300398    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f25887-41c2-498f-87c9-2299fa9008c2-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:19.300438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.300406    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f25887-41c2-498f-87c9-2299fa9008c2-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:19.300438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.300415    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtx5\" (UniqueName: \"kubernetes.io/projected/75f25887-41c2-498f-87c9-2299fa9008c2-kube-api-access-cgtx5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:19.504281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.504178    4519 generic.go:356] "Generic (PLEG): container finished" podID="75f25887-41c2-498f-87c9-2299fa9008c2" containerID="bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622" exitCode=0
Oct 30 20:17:19.504281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.504242    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"
Oct 30 20:17:19.504587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.504386    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x" event={"ID":"75f25887-41c2-498f-87c9-2299fa9008c2","Type":"ContainerDied","Data":"bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622"}
Oct 30 20:17:19.504587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.504425    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x" event={"ID":"75f25887-41c2-498f-87c9-2299fa9008c2","Type":"ContainerDied","Data":"f968af5d10edcbae387fca3cb4642918828f717e6d76562618798ce63a343763"}
Oct 30 20:17:19.504587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.504446    4519 scope.go:122] "RemoveContainer" containerID="bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622"
Oct 30 20:17:19.519513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.519458    4519 scope.go:122] "RemoveContainer" containerID="bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622"
Oct 30 20:17:19.519856 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:17:19.519832    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622\": container with ID starting with bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622 not found: ID does not exist" containerID="bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622"
Oct 30 20:17:19.519920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.519870    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622"} err="failed to get container status \"bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622\": rpc error: code = NotFound desc = could not find container \"bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622\": container with ID starting with bfd6b68b79889efbc5180f46d9c6b7c93bc3b448ae540d6b4198e7522611c622 not found: ID does not exist"
Oct 30 20:17:19.568592 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.568522    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"]
Oct 30 20:17:19.581522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.579046    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c8fa69/auth-test-model-c8fa69-kserve-56f96655b9-pr46x"]
Oct 30 20:17:19.613201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:19.613160    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f25887-41c2-498f-87c9-2299fa9008c2" path="/var/lib/kubelet/pods/75f25887-41c2-498f-87c9-2299fa9008c2/volumes"
Oct 30 20:17:22.200023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.199958    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"]
Oct 30 20:17:22.201600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.200800    4519 cpu_manager.go:414] "containerMap: removing container" podUID="415a3dfe-bd6e-4878-a940-c1b34360ccef" containerName="revoke-keys"
Oct 30 20:17:22.201600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.200825    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415a3dfe-bd6e-4878-a940-c1b34360ccef" containerName="revoke-keys"
Oct 30 20:17:22.201600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.200863    4519 cpu_manager.go:414] "containerMap: removing container" podUID="75f25887-41c2-498f-87c9-2299fa9008c2" containerName="main"
Oct 30 20:17:22.201600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.200871    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f25887-41c2-498f-87c9-2299fa9008c2" containerName="main"
Oct 30 20:17:22.201600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.201056    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="415a3dfe-bd6e-4878-a940-c1b34360ccef" containerName="revoke-keys"
Oct 30 20:17:22.201600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.201072    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f25887-41c2-498f-87c9-2299fa9008c2" containerName="main"
Oct 30 20:17:22.210496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.209897    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.232576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.232542    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"]
Oct 30 20:17:22.239745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.238351    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-188d45-openshift-default-dockercfg-tqmpj\""
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.507927    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nm2s\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-kube-api-access-9nm2s\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508074    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-credential-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508107    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-certs\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508178    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508212    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-data\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508257    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85027075-443e-4323-a372-86b86a6dd098-istio-podinfo\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508298    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85027075-443e-4323-a372-86b86a6dd098-istiod-ca-cert\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508397    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-istio-token\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.508546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.508445    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-envoy\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.621276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.610432    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-credential-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.621876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.621582    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-certs\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.635496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.611916    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-credential-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.635743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.626054    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-certs\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.635897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.629674    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.636033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.636017    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-data\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.642925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.642900    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85027075-443e-4323-a372-86b86a6dd098-istio-podinfo\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.651581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.651555    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85027075-443e-4323-a372-86b86a6dd098-istiod-ca-cert\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.657644 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.657618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-istio-token\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.661496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.658109    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-envoy\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.672716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.672690    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nm2s\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-kube-api-access-9nm2s\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.672974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.630049    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.673097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.642814    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-data\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.673194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.652410    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85027075-443e-4323-a372-86b86a6dd098-istiod-ca-cert\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.689487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.675584    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85027075-443e-4323-a372-86b86a6dd098-istio-podinfo\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.704442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.704400    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-istio-token\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.704792 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.704769    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-envoy\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.708374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.708343    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nm2s\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-kube-api-access-9nm2s\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:22.839372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:22.839285    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:23.272950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.272906    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:23.273499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.273454    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:23.273663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.273644    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:23.280611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.279820    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"]
Oct 30 20:17:23.562729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.562640    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" event={"ID":"85027075-443e-4323-a372-86b86a6dd098","Type":"ContainerStarted","Data":"b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451"}
Oct 30 20:17:23.562729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.562698    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" event={"ID":"85027075-443e-4323-a372-86b86a6dd098","Type":"ContainerStarted","Data":"0b6ea8033572d1b2b708940ec38543afad7b9908a2f7a0baebd77d7dff882a1c"}
Oct 30 20:17:23.607941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.607868    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podStartSLOduration=1.6078460140000002 podStartE2EDuration="1.607846014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:23.606966115 +0000 UTC m=+2568.107669838" watchObservedRunningTime="2026-10-30 20:17:23.607846014 +0000 UTC m=+2568.108549739"
Oct 30 20:17:23.840142 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:23.840049    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:24.849559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:24.848384    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:25.589059 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:25.589029    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:25.590048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:25.590024    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:17:25.991486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:25.979787    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"]
Oct 30 20:17:26.030491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.029780    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"]
Oct 30 20:17:26.030491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.029914    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.064302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.064267    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9d2720-openshift-default-dockercfg-nhcwg\""
Oct 30 20:17:26.394073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.393976    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-envoy\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394043    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-podinfo\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394168    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a08deb-6080-41c4-baea-37afb9d69f36-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394208    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78d7\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-kube-api-access-r78d7\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394320    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-token\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394373    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-credential-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394482    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-data\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394575    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.394712 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.394604    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-certs\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.495927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.495893    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a08deb-6080-41c4-baea-37afb9d69f36-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.495944    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78d7\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-kube-api-access-r78d7\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.495994    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-token\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.496030    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-credential-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.496128    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-data\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.496216    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.496245    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-certs\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.496374    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-envoy\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.496448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.496420    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-podinfo\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.497021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.496993    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-credential-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.497269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.497248    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-certs\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.497572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.497552    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a08deb-6080-41c4-baea-37afb9d69f36-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.497866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.497837    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.498004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.497983    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-data\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.499713 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.499686    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-envoy\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.499791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.499711    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-podinfo\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.536488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.536291    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-token\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.544486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.541353    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78d7\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-kube-api-access-r78d7\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.738436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.738280    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:26.753020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.752657    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:17:26.753020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:26.752715    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:17:27.104781 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:17:27.104749    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a08deb_6080_41c4_baea_37afb9d69f36.slice/crio-eba5ffeca949906fde34642ca627f81e9b15c218694179398ee801f84015aba2 WatchSource:0}: Error finding container eba5ffeca949906fde34642ca627f81e9b15c218694179398ee801f84015aba2: Status 404 returned error can't find the container with id eba5ffeca949906fde34642ca627f81e9b15c218694179398ee801f84015aba2
Oct 30 20:17:27.106371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.106341    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"]
Oct 30 20:17:27.108540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.108508    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:27.108652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.108584    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:27.108652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.108631    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:27.665636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.665534    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" event={"ID":"c6a08deb-6080-41c4-baea-37afb9d69f36","Type":"ContainerStarted","Data":"b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982"}
Oct 30 20:17:27.665636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.665599    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" event={"ID":"c6a08deb-6080-41c4-baea-37afb9d69f36","Type":"ContainerStarted","Data":"eba5ffeca949906fde34642ca627f81e9b15c218694179398ee801f84015aba2"}
Oct 30 20:17:27.710093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.710034    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podStartSLOduration=2.71001415 podStartE2EDuration="2.71001415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:27.707098651 +0000 UTC m=+2572.207802375" watchObservedRunningTime="2026-10-30 20:17:27.71001415 +0000 UTC m=+2572.210717875"
Oct 30 20:17:27.742591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:27.742558    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:28.677500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:28.677423    4519 generic.go:356] "Generic (PLEG): container finished" podID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerID="604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef" exitCode=0
Oct 30 20:17:28.677961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:28.677527    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" event={"ID":"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3","Type":"ContainerDied","Data":"604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef"}
Oct 30 20:17:28.678499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:28.678450    4519 scope.go:122] "RemoveContainer" containerID="604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef"
Oct 30 20:17:28.680634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:28.680599    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:28.680744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:28.680688    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:28.680788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:28.680738    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:28.751774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:28.751744    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:29.730074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:29.730027    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" event={"ID":"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3","Type":"ContainerStarted","Data":"99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477"}
Oct 30 20:17:29.731083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:29.730335    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:29.731322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:29.731304    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:17:30.288925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:30.288845    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:17:30.289168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:30.289140    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 20:17:30.289244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:30.289205    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 20:17:31.289534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:31.289494    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 20:17:31.289961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:31.289555    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 20:17:32.304487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:32.290874    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 20:17:32.304487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:32.290936    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 20:17:32.407523 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:17:32.404666    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:17:33.288979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:33.288937    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 20:17:33.289164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:33.289023    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 20:17:34.289245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:34.289203    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 20:17:34.289721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:34.289265    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 20:17:35.289914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:35.289863    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 20:17:35.290421 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:35.289927    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 20:17:36.295512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:36.295442    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:17:36.295944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:36.295931    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:17:36.296614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:36.296594    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:17:40.038491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.036200    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"]
Oct 30 20:17:40.040812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.040784    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.067334    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.067437    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.067482    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzwp\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-kube-api-access-fqzwp\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.069247    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1781aa1d-20cb-44a8-b7f8-0004676efc46-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.069325    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.069404    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.069498    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.069593    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.072489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.069631    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.082483 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.080286    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-5c875d-openshift-default-dockercfg-crmft\""
Oct 30 20:17:40.088322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.084253    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"]
Oct 30 20:17:40.211491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.186279    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.213900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.213867    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.237490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.214175    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.237770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.237746    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.238317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.238294    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.238873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.238849    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.239312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.239292    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzwp\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-kube-api-access-fqzwp\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.239831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.239810    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1781aa1d-20cb-44a8-b7f8-0004676efc46-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.252487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.240838    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.252784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.238176    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.252918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.239241    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.253025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.222780    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.253133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.236020    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.253239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.240732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1781aa1d-20cb-44a8-b7f8-0004676efc46-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.253340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.238722    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.253439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.245815    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.279195 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.279166    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzwp\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-kube-api-access-fqzwp\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.279893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.279870    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.372134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.372037    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:40.843027 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:17:40.842990    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1781aa1d_20cb_44a8_b7f8_0004676efc46.slice/crio-9cce90c4e940e914f804a0c0254c884ade8610bdde5e38766241a1fcc73cc88a WatchSource:0}: Error finding container 9cce90c4e940e914f804a0c0254c884ade8610bdde5e38766241a1fcc73cc88a: Status 404 returned error can't find the container with id 9cce90c4e940e914f804a0c0254c884ade8610bdde5e38766241a1fcc73cc88a
Oct 30 20:17:40.844578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.844549    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"]
Oct 30 20:17:40.846015 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.845982    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:40.846094 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.846052    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:40.846094 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.846079    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:40.882547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:40.882508    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" event={"ID":"1781aa1d-20cb-44a8-b7f8-0004676efc46","Type":"ContainerStarted","Data":"9cce90c4e940e914f804a0c0254c884ade8610bdde5e38766241a1fcc73cc88a"}
Oct 30 20:17:41.895086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:41.895044    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" event={"ID":"1781aa1d-20cb-44a8-b7f8-0004676efc46","Type":"ContainerStarted","Data":"3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46"}
Oct 30 20:17:41.962565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:41.962508    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" podStartSLOduration=1.962492602 podStartE2EDuration="1.962492602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:41.948862303 +0000 UTC m=+2586.449566028" watchObservedRunningTime="2026-10-30 20:17:41.962492602 +0000 UTC m=+2586.463196326"
Oct 30 20:17:42.374076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.374032    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:42.382903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.382867    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:42.730440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.727150    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"]
Oct 30 20:17:42.739447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.739411    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:42.750594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.750559    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"]
Oct 30 20:17:42.754287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.754257    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:42.754427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.754409    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:17:42.902921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.902885    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:42.903401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.903024    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:42.903401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.903143    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzxm\" (UniqueName: \"kubernetes.io/projected/21ad26ac-3997-441f-9f79-5c4253e128a7-kube-api-access-8nzxm\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:42.904769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.904749    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:42.905943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:42.905920    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:17:43.005340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.005250    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:43.005562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.005391    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:43.005562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.005444    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzxm\" (UniqueName: \"kubernetes.io/projected/21ad26ac-3997-441f-9f79-5c4253e128a7-kube-api-access-8nzxm\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:43.008780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.008747    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:43.031202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.031171    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:43.034454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.034426    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzxm\" (UniqueName: \"kubernetes.io/projected/21ad26ac-3997-441f-9f79-5c4253e128a7-kube-api-access-8nzxm\") pod \"maas-api-revoke-sub-51a4035d-2ltdl\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:43.064449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.064423    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:43.562055 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.562019    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"]
Oct 30 20:17:43.913247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:43.913219    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl" event={"ID":"21ad26ac-3997-441f-9f79-5c4253e128a7","Type":"ContainerStarted","Data":"0c9a3493d8200c85b9db3eeee591ce43d76fd7ddb22ce76ab48e6d914b5dd2f1"}
Oct 30 20:17:44.054285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:44.054249    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"]
Oct 30 20:17:44.064516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:44.064483    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c8fc40-zrvqh"]
Oct 30 20:17:44.195616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:44.195589    4519 scope.go:122] "RemoveContainer" containerID="53146a4809f631b86dd944b07743b936c6ef5e3a937ac07ea3625929477d59af"
Oct 30 20:17:44.206396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:44.206376    4519 scope.go:122] "RemoveContainer" containerID="dde2d9e263258b98f55487df3f5ac49847f22047818aedd1b6302b1ea6ff2476"
Oct 30 20:17:44.924274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:44.924236    4519 generic.go:356] "Generic (PLEG): container finished" podID="21ad26ac-3997-441f-9f79-5c4253e128a7" containerID="fdbe96550442cfc40adda5248941239edfdf4d06ca1c83450daa495042e1d00b" exitCode=0
Oct 30 20:17:44.924755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:44.924318    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl" event={"ID":"21ad26ac-3997-441f-9f79-5c4253e128a7","Type":"ContainerDied","Data":"fdbe96550442cfc40adda5248941239edfdf4d06ca1c83450daa495042e1d00b"}
Oct 30 20:17:45.613012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:45.612975    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415a3dfe-bd6e-4878-a940-c1b34360ccef" path="/var/lib/kubelet/pods/415a3dfe-bd6e-4878-a940-c1b34360ccef/volumes"
Oct 30 20:17:46.131894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.131826    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:46.209586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.209461    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ad26ac-3997-441f-9f79-5c4253e128a7-kube-api-access-8nzxm\" (UniqueName: \"kubernetes.io/projected/21ad26ac-3997-441f-9f79-5c4253e128a7-kube-api-access-8nzxm\") pod \"21ad26ac-3997-441f-9f79-5c4253e128a7\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") "
Oct 30 20:17:46.209586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.209578    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-xks-ca\") pod \"21ad26ac-3997-441f-9f79-5c4253e128a7\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") "
Oct 30 20:17:46.209830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.209621    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-openshift-ca\") pod \"21ad26ac-3997-441f-9f79-5c4253e128a7\" (UID: \"21ad26ac-3997-441f-9f79-5c4253e128a7\") "
Oct 30 20:17:46.210174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.210137    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-openshift-ca" pod "21ad26ac-3997-441f-9f79-5c4253e128a7" (UID: "21ad26ac-3997-441f-9f79-5c4253e128a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:46.210318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.210271    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:46.212406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.212136    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ad26ac-3997-441f-9f79-5c4253e128a7-kube-api-access-8nzxm" pod "21ad26ac-3997-441f-9f79-5c4253e128a7" (UID: "21ad26ac-3997-441f-9f79-5c4253e128a7"). InnerVolumeSpecName "kube-api-access-8nzxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:46.215808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.215776    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-xks-ca" pod "21ad26ac-3997-441f-9f79-5c4253e128a7" (UID: "21ad26ac-3997-441f-9f79-5c4253e128a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:46.311536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.311502    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nzxm\" (UniqueName: \"kubernetes.io/projected/21ad26ac-3997-441f-9f79-5c4253e128a7-kube-api-access-8nzxm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:46.311536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.311533    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ad26ac-3997-441f-9f79-5c4253e128a7-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:46.941959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.941920    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl" event={"ID":"21ad26ac-3997-441f-9f79-5c4253e128a7","Type":"ContainerDied","Data":"0c9a3493d8200c85b9db3eeee591ce43d76fd7ddb22ce76ab48e6d914b5dd2f1"}
Oct 30 20:17:46.941959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.941946    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"
Oct 30 20:17:46.942186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:46.941977    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9a3493d8200c85b9db3eeee591ce43d76fd7ddb22ce76ab48e6d914b5dd2f1"
Oct 30 20:17:50.967067 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:50.967032    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"]
Oct 30 20:17:50.967613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:50.967316    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy" containerID="cri-o://0970ee28848e15352cf9aed9479408d376a8fe78b50c5b4a4e84baa90aafd290" gracePeriod=30
Oct 30 20:17:51.632973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.632943    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"]
Oct 30 20:17:51.633543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.633526    4519 cpu_manager.go:414] "containerMap: removing container" podUID="21ad26ac-3997-441f-9f79-5c4253e128a7" containerName="revoke-keys"
Oct 30 20:17:51.633612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.633543    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ad26ac-3997-441f-9f79-5c4253e128a7" containerName="revoke-keys"
Oct 30 20:17:51.633692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.633642    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ad26ac-3997-441f-9f79-5c4253e128a7" containerName="revoke-keys"
Oct 30 20:17:51.637360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.637341    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.658183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.658146    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"]
Oct 30 20:17:51.662580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.662559    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:51.663096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.663074    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:17:51.730858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.730822    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.731004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.730874    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6mj\" (UniqueName: \"kubernetes.io/projected/4627d467-37c0-4a50-bd4c-d73781f93b24-kube-api-access-gm6mj\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.731044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.731028    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.832258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.832218    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.832500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.832302    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6mj\" (UniqueName: \"kubernetes.io/projected/4627d467-37c0-4a50-bd4c-d73781f93b24-kube-api-access-gm6mj\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.832500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.832450    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.833158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.833134    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.834966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.834946    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.852006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.851977    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6mj\" (UniqueName: \"kubernetes.io/projected/4627d467-37c0-4a50-bd4c-d73781f93b24-kube-api-access-gm6mj\") pod \"maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:51.952585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:51.952546    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:52.276306 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:17:52.276267    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4627d467_37c0_4a50_bd4c_d73781f93b24.slice/crio-131bd343d9fe30d89ff84da2e7231a9f139b0aaaed18080c42d1f321a0600227 WatchSource:0}: Error finding container 131bd343d9fe30d89ff84da2e7231a9f139b0aaaed18080c42d1f321a0600227: Status 404 returned error can't find the container with id 131bd343d9fe30d89ff84da2e7231a9f139b0aaaed18080c42d1f321a0600227
Oct 30 20:17:52.279811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:52.279787    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"]
Oct 30 20:17:52.490126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:52.490091    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"]
Oct 30 20:17:53.175890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:53.175562    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2" event={"ID":"4627d467-37c0-4a50-bd4c-d73781f93b24","Type":"ContainerStarted","Data":"915ec22dd49106bef7bb322a678a510b1b0a929f206cf1d157e5e38e86afd4fc"}
Oct 30 20:17:53.175890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:53.175629    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2" event={"ID":"4627d467-37c0-4a50-bd4c-d73781f93b24","Type":"ContainerStarted","Data":"131bd343d9fe30d89ff84da2e7231a9f139b0aaaed18080c42d1f321a0600227"}
Oct 30 20:17:54.189764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:54.189724    4519 generic.go:356] "Generic (PLEG): container finished" podID="4627d467-37c0-4a50-bd4c-d73781f93b24" containerID="915ec22dd49106bef7bb322a678a510b1b0a929f206cf1d157e5e38e86afd4fc" exitCode=0
Oct 30 20:17:54.190279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:54.189818    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2" event={"ID":"4627d467-37c0-4a50-bd4c-d73781f93b24","Type":"ContainerDied","Data":"915ec22dd49106bef7bb322a678a510b1b0a929f206cf1d157e5e38e86afd4fc"}
Oct 30 20:17:54.925560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:54.925527    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"]
Oct 30 20:17:54.930281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:54.930254    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:54.958511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:54.955868    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"]
Oct 30 20:17:55.008280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.008231    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.008750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.008717    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8258\" (UniqueName: \"kubernetes.io/projected/c5387407-2f32-4cdc-83c4-fe88663933db-kube-api-access-g8258\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.008975 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.008954    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.110405 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.110363    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.110622 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.110458    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.110622 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.110609    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8258\" (UniqueName: \"kubernetes.io/projected/c5387407-2f32-4cdc-83c4-fe88663933db-kube-api-access-g8258\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.111098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.111073    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.112990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.112967    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.129806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.129780    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8258\" (UniqueName: \"kubernetes.io/projected/c5387407-2f32-4cdc-83c4-fe88663933db-kube-api-access-g8258\") pod \"maas-api-revoke-sub-d4dd013d-gcdn9\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.248365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.248323    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:55.389084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.387400    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:55.414737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.414709    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-xks-ca\") pod \"4627d467-37c0-4a50-bd4c-d73781f93b24\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") "
Oct 30 20:17:55.415253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.415024    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-openshift-ca\") pod \"4627d467-37c0-4a50-bd4c-d73781f93b24\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") "
Oct 30 20:17:55.415253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.415074    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4627d467-37c0-4a50-bd4c-d73781f93b24-kube-api-access-gm6mj\" (UniqueName: \"kubernetes.io/projected/4627d467-37c0-4a50-bd4c-d73781f93b24-kube-api-access-gm6mj\") pod \"4627d467-37c0-4a50-bd4c-d73781f93b24\" (UID: \"4627d467-37c0-4a50-bd4c-d73781f93b24\") "
Oct 30 20:17:55.415511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.415438    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-openshift-ca" pod "4627d467-37c0-4a50-bd4c-d73781f93b24" (UID: "4627d467-37c0-4a50-bd4c-d73781f93b24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:55.415786 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.415758    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:55.417381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.417352    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4627d467-37c0-4a50-bd4c-d73781f93b24-kube-api-access-gm6mj" pod "4627d467-37c0-4a50-bd4c-d73781f93b24" (UID: "4627d467-37c0-4a50-bd4c-d73781f93b24"). InnerVolumeSpecName "kube-api-access-gm6mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:55.418034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.418009    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-xks-ca" pod "4627d467-37c0-4a50-bd4c-d73781f93b24" (UID: "4627d467-37c0-4a50-bd4c-d73781f93b24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:55.517191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.517116    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6mj\" (UniqueName: \"kubernetes.io/projected/4627d467-37c0-4a50-bd4c-d73781f93b24-kube-api-access-gm6mj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:55.517191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.517146    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4627d467-37c0-4a50-bd4c-d73781f93b24-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:55.580286 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:17:55.580253    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5387407_2f32_4cdc_83c4_fe88663933db.slice/crio-86ef998b90586e3423a1ef8c5e89fab632eca7dc13c9506887748f1f5677e8c9 WatchSource:0}: Error finding container 86ef998b90586e3423a1ef8c5e89fab632eca7dc13c9506887748f1f5677e8c9: Status 404 returned error can't find the container with id 86ef998b90586e3423a1ef8c5e89fab632eca7dc13c9506887748f1f5677e8c9
Oct 30 20:17:55.583636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:55.583588    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"]
Oct 30 20:17:56.211403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.211370    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9" event={"ID":"c5387407-2f32-4cdc-83c4-fe88663933db","Type":"ContainerStarted","Data":"a246f4ba9cde4925b2e1f7efaec03d978b4d982ccfd2c3faa1980a010edd4831"}
Oct 30 20:17:56.211577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.211424    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9" event={"ID":"c5387407-2f32-4cdc-83c4-fe88663933db","Type":"ContainerStarted","Data":"86ef998b90586e3423a1ef8c5e89fab632eca7dc13c9506887748f1f5677e8c9"}
Oct 30 20:17:56.214144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.213496    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2" event={"ID":"4627d467-37c0-4a50-bd4c-d73781f93b24","Type":"ContainerDied","Data":"131bd343d9fe30d89ff84da2e7231a9f139b0aaaed18080c42d1f321a0600227"}
Oct 30 20:17:56.214144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.213519    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"
Oct 30 20:17:56.214144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.213552    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="131bd343d9fe30d89ff84da2e7231a9f139b0aaaed18080c42d1f321a0600227"
Oct 30 20:17:56.215908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.215883    4519 generic.go:356] "Generic (PLEG): container finished" podID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerID="0970ee28848e15352cf9aed9479408d376a8fe78b50c5b4a4e84baa90aafd290" exitCode=0
Oct 30 20:17:56.216241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.216109    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" event={"ID":"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e","Type":"ContainerDied","Data":"0970ee28848e15352cf9aed9479408d376a8fe78b50c5b4a4e84baa90aafd290"}
Oct 30 20:17:56.216241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.216149    4519 scope.go:122] "RemoveContainer" containerID="da9cf93be5dba0da7c8924357effb264738b1470d96d8cc910d42f9bfa31f2e1"
Oct 30 20:17:56.253204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.253026    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9" podStartSLOduration=2.253008921 podStartE2EDuration="2.253008921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:56.252752243 +0000 UTC m=+2600.753455979" watchObservedRunningTime="2026-10-30 20:17:56.253008921 +0000 UTC m=+2600.753712648"
Oct 30 20:17:56.294022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.294002    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:17:56.327843 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.327817    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-credential-socket\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.327985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.327911    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-token\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.327985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.327933    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-data\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.327985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.327975    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-envoy\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.328165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328056    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-certs\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.328165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328108    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istiod-ca-cert\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.328165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328138    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-podinfo\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.328165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328132    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-credential-socket" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:56.328399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328194    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-socket\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.328399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328242    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-data" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:56.328399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328373    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-kube-api-access-k2fjk\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-kube-api-access-k2fjk\") pod \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\" (UID: \"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e\") "
Oct 30 20:17:56.328399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328377    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-certs" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:56.328735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328591    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-socket" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:56.328735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.328638    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istiod-ca-cert" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:56.329272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.329250    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.329272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.329271    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.329423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.329280    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.329423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.329289    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.329423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.329297    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.330761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.330738    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-envoy" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:56.330857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.330767    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-token" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:56.330857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.330770    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-podinfo" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:56.330990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.330879    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-kube-api-access-k2fjk" pod "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" (UID: "0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e"). InnerVolumeSpecName "kube-api-access-k2fjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:56.430716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.430681    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.430716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.430712    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.430716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.430722    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:56.430946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:56.430731    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fjk\" (UniqueName: \"kubernetes.io/projected/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e-kube-api-access-k2fjk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:57.226396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.226359    4519 generic.go:356] "Generic (PLEG): container finished" podID="c5387407-2f32-4cdc-83c4-fe88663933db" containerID="a246f4ba9cde4925b2e1f7efaec03d978b4d982ccfd2c3faa1980a010edd4831" exitCode=0
Oct 30 20:17:57.226572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.226431    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9" event={"ID":"c5387407-2f32-4cdc-83c4-fe88663933db","Type":"ContainerDied","Data":"a246f4ba9cde4925b2e1f7efaec03d978b4d982ccfd2c3faa1980a010edd4831"}
Oct 30 20:17:57.227968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.227934    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx" event={"ID":"0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e","Type":"ContainerDied","Data":"d46a9456aca1d963add695badacf8de63f98aaa2f99a9501740a3fca373fe19f"}
Oct 30 20:17:57.227968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.227962    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"
Oct 30 20:17:57.228133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.227999    4519 scope.go:122] "RemoveContainer" containerID="0970ee28848e15352cf9aed9479408d376a8fe78b50c5b4a4e84baa90aafd290"
Oct 30 20:17:57.306506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.306432    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"]
Oct 30 20:17:57.312412 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.312384    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c8fc40-openshift-default-75d78599c9-8fgtx"]
Oct 30 20:17:57.613763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:57.613726    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" path="/var/lib/kubelet/pods/0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e/volumes"
Oct 30 20:17:58.407635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.407606    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:58.466056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.466019    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-openshift-ca\") pod \"c5387407-2f32-4cdc-83c4-fe88663933db\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") "
Oct 30 20:17:58.466257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.466182    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5387407-2f32-4cdc-83c4-fe88663933db-kube-api-access-g8258\" (UniqueName: \"kubernetes.io/projected/c5387407-2f32-4cdc-83c4-fe88663933db-kube-api-access-g8258\") pod \"c5387407-2f32-4cdc-83c4-fe88663933db\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") "
Oct 30 20:17:58.466341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.466321    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-xks-ca\") pod \"c5387407-2f32-4cdc-83c4-fe88663933db\" (UID: \"c5387407-2f32-4cdc-83c4-fe88663933db\") "
Oct 30 20:17:58.466516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.466455    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-openshift-ca" pod "c5387407-2f32-4cdc-83c4-fe88663933db" (UID: "c5387407-2f32-4cdc-83c4-fe88663933db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:58.467024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.466995    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:58.471519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.469305    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5387407-2f32-4cdc-83c4-fe88663933db-kube-api-access-g8258" pod "c5387407-2f32-4cdc-83c4-fe88663933db" (UID: "c5387407-2f32-4cdc-83c4-fe88663933db"). InnerVolumeSpecName "kube-api-access-g8258". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.477623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.477595    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-xks-ca" pod "c5387407-2f32-4cdc-83c4-fe88663933db" (UID: "c5387407-2f32-4cdc-83c4-fe88663933db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:58.569177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.569104    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8258\" (UniqueName: \"kubernetes.io/projected/c5387407-2f32-4cdc-83c4-fe88663933db-kube-api-access-g8258\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:58.569177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:58.569144    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5387407-2f32-4cdc-83c4-fe88663933db-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:17:59.246671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:59.246641    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"
Oct 30 20:17:59.246909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:59.246682    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9" event={"ID":"c5387407-2f32-4cdc-83c4-fe88663933db","Type":"ContainerDied","Data":"86ef998b90586e3423a1ef8c5e89fab632eca7dc13c9506887748f1f5677e8c9"}
Oct 30 20:17:59.246909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:17:59.246739    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ef998b90586e3423a1ef8c5e89fab632eca7dc13c9506887748f1f5677e8c9"
Oct 30 20:18:02.223020 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:02.222980    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:18:07.527506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.527421    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9" podUID="6d959bec-39ca-41c7-b751-c146392a88ef" containerName="main" containerID="cri-o://2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0" gracePeriod=45
Oct 30 20:18:07.573886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.573850    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9" podUID="6d959bec-39ca-41c7-b751-c146392a88ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.252:8000/ready\": dial tcp 10.133.0.252:8000: connect: connection refused"
Oct 30 20:18:07.845811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.845769    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:18:07.901164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.901126    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d959bec-39ca-41c7-b751-c146392a88ef-kube-api-access-ndn84\" (UniqueName: \"kubernetes.io/projected/6d959bec-39ca-41c7-b751-c146392a88ef-kube-api-access-ndn84\") pod \"6d959bec-39ca-41c7-b751-c146392a88ef\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") "
Oct 30 20:18:07.901398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.901183    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-dshm\") pod \"6d959bec-39ca-41c7-b751-c146392a88ef\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") "
Oct 30 20:18:07.901398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.901292    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-tmp-dir\") pod \"6d959bec-39ca-41c7-b751-c146392a88ef\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") "
Oct 30 20:18:07.901729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.901427    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-model-cache\") pod \"6d959bec-39ca-41c7-b751-c146392a88ef\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") "
Oct 30 20:18:07.901729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.901570    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-home\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-home\") pod \"6d959bec-39ca-41c7-b751-c146392a88ef\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") "
Oct 30 20:18:07.901729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.901618    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d959bec-39ca-41c7-b751-c146392a88ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d959bec-39ca-41c7-b751-c146392a88ef-tls-certs\") pod \"6d959bec-39ca-41c7-b751-c146392a88ef\" (UID: \"6d959bec-39ca-41c7-b751-c146392a88ef\") "
Oct 30 20:18:07.903438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.903206    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-model-cache" pod "6d959bec-39ca-41c7-b751-c146392a88ef" (UID: "6d959bec-39ca-41c7-b751-c146392a88ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:07.903438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.903313    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-home" pod "6d959bec-39ca-41c7-b751-c146392a88ef" (UID: "6d959bec-39ca-41c7-b751-c146392a88ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:07.903763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.903734    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-tmp-dir" pod "6d959bec-39ca-41c7-b751-c146392a88ef" (UID: "6d959bec-39ca-41c7-b751-c146392a88ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:07.921136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.920097    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d959bec-39ca-41c7-b751-c146392a88ef-tls-certs" pod "6d959bec-39ca-41c7-b751-c146392a88ef" (UID: "6d959bec-39ca-41c7-b751-c146392a88ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:07.921136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.920122    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d959bec-39ca-41c7-b751-c146392a88ef-kube-api-access-ndn84" pod "6d959bec-39ca-41c7-b751-c146392a88ef" (UID: "6d959bec-39ca-41c7-b751-c146392a88ef"). InnerVolumeSpecName "kube-api-access-ndn84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:07.921600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:07.921577    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-dshm" pod "6d959bec-39ca-41c7-b751-c146392a88ef" (UID: "6d959bec-39ca-41c7-b751-c146392a88ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:08.003737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.003697    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:08.003737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.003729    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:08.003737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.003742    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:08.004000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.003753    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d959bec-39ca-41c7-b751-c146392a88ef-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:08.004000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.003765    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndn84\" (UniqueName: \"kubernetes.io/projected/6d959bec-39ca-41c7-b751-c146392a88ef-kube-api-access-ndn84\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:08.004000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.003779    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d959bec-39ca-41c7-b751-c146392a88ef-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:08.334216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.334087    4519 generic.go:356] "Generic (PLEG): container finished" podID="6d959bec-39ca-41c7-b751-c146392a88ef" containerID="2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0" exitCode=0
Oct 30 20:18:08.334216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.334191    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9" event={"ID":"6d959bec-39ca-41c7-b751-c146392a88ef","Type":"ContainerDied","Data":"2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0"}
Oct 30 20:18:08.334439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.334234    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9" event={"ID":"6d959bec-39ca-41c7-b751-c146392a88ef","Type":"ContainerDied","Data":"b3c1b59592b06bff0c09153295934e594d6b505c0681f2d47fe061658aaaa662"}
Oct 30 20:18:08.334439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.334254    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"
Oct 30 20:18:08.334439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.334266    4519 scope.go:122] "RemoveContainer" containerID="2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0"
Oct 30 20:18:08.345378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.345358    4519 scope.go:122] "RemoveContainer" containerID="2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0"
Oct 30 20:18:08.345649 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:08.345630    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0\": container with ID starting with 2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0 not found: ID does not exist" containerID="2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0"
Oct 30 20:18:08.345717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.345657    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0"} err="failed to get container status \"2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0\": rpc error: code = NotFound desc = could not find container \"2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0\": container with ID starting with 2790b86c2e452218808f06cd4b3744e15624c9cbd7fe7bc8b0bb8d8abca749b0 not found: ID does not exist"
Oct 30 20:18:08.391918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.391876    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"]
Oct 30 20:18:08.396266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:08.396236    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/auth-test-model-5c2e32-kserve-699d6ddc89-22nt9"]
Oct 30 20:18:09.675061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:09.675016    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d959bec-39ca-41c7-b751-c146392a88ef" path="/var/lib/kubelet/pods/6d959bec-39ca-41c7-b751-c146392a88ef/volumes"
Oct 30 20:18:17.941319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:17.941261    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"]
Oct 30 20:18:21.844649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.844612    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"]
Oct 30 20:18:21.845200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845184    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4627d467-37c0-4a50-bd4c-d73781f93b24" containerName="revoke-keys"
Oct 30 20:18:21.845247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845201    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4627d467-37c0-4a50-bd4c-d73781f93b24" containerName="revoke-keys"
Oct 30 20:18:21.845247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845214    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6d959bec-39ca-41c7-b751-c146392a88ef" containerName="main"
Oct 30 20:18:21.845247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845219    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d959bec-39ca-41c7-b751-c146392a88ef" containerName="main"
Oct 30 20:18:21.845247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845231    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy"
Oct 30 20:18:21.845247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845237    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845260    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845266    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845279    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c5387407-2f32-4cdc-83c4-fe88663933db" containerName="revoke-keys"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845284    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5387407-2f32-4cdc-83c4-fe88663933db" containerName="revoke-keys"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845369    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845379    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5387407-2f32-4cdc-83c4-fe88663933db" containerName="revoke-keys"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845388    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4627d467-37c0-4a50-bd4c-d73781f93b24" containerName="revoke-keys"
Oct 30 20:18:21.845414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.845394    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d959bec-39ca-41c7-b751-c146392a88ef" containerName="main"
Oct 30 20:18:21.853485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.850983    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:21.871849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.871817    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"]
Oct 30 20:18:21.874900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.874876    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:21.875056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.874973    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:18:21.945306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.945261    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:21.945642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.945617    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5k6\" (UniqueName: \"kubernetes.io/projected/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-kube-api-access-tc5k6\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:21.945832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:21.945816    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.047410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.047373    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.047602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.047447    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5k6\" (UniqueName: \"kubernetes.io/projected/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-kube-api-access-tc5k6\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.047705 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.047595    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.048391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.048361    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.049868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.049842    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.069446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.069418    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5k6\" (UniqueName: \"kubernetes.io/projected/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-kube-api-access-tc5k6\") pod \"maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.171928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.171894    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:22.485427 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:18:22.485385    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf92ee67_60cf_4e97_bf42_b4c1e13ec70e.slice/crio-a2dda079505189194ccf984547b5452ce4050a2141337ca8a283943c5c381596 WatchSource:0}: Error finding container a2dda079505189194ccf984547b5452ce4050a2141337ca8a283943c5c381596: Status 404 returned error can't find the container with id a2dda079505189194ccf984547b5452ce4050a2141337ca8a283943c5c381596
Oct 30 20:18:22.487057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:22.486988    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"]
Oct 30 20:18:23.466994 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:23.466949    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr" event={"ID":"af92ee67-60cf-4e97-bf42-b4c1e13ec70e","Type":"ContainerStarted","Data":"f2f4ac3f20c967991272834d94dd4b02126005acdf84cdf7356c89409ddd651e"}
Oct 30 20:18:23.467500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:23.467027    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr" event={"ID":"af92ee67-60cf-4e97-bf42-b4c1e13ec70e","Type":"ContainerStarted","Data":"a2dda079505189194ccf984547b5452ce4050a2141337ca8a283943c5c381596"}
Oct 30 20:18:24.477669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:24.477627    4519 generic.go:356] "Generic (PLEG): container finished" podID="af92ee67-60cf-4e97-bf42-b4c1e13ec70e" containerID="f2f4ac3f20c967991272834d94dd4b02126005acdf84cdf7356c89409ddd651e" exitCode=0
Oct 30 20:18:24.478089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:24.477705    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr" event={"ID":"af92ee67-60cf-4e97-bf42-b4c1e13ec70e","Type":"ContainerDied","Data":"f2f4ac3f20c967991272834d94dd4b02126005acdf84cdf7356c89409ddd651e"}
Oct 30 20:18:25.664427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.664394    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:25.733578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.733544    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-kube-api-access-tc5k6\" (UniqueName: \"kubernetes.io/projected/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-kube-api-access-tc5k6\") pod \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") "
Oct 30 20:18:25.733818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.733802    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-xks-ca\") pod \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") "
Oct 30 20:18:25.733911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.733839    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-openshift-ca\") pod \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\" (UID: \"af92ee67-60cf-4e97-bf42-b4c1e13ec70e\") "
Oct 30 20:18:25.734326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.734296    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-openshift-ca" pod "af92ee67-60cf-4e97-bf42-b4c1e13ec70e" (UID: "af92ee67-60cf-4e97-bf42-b4c1e13ec70e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:25.734584 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.734563    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:25.736378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.736338    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-kube-api-access-tc5k6" pod "af92ee67-60cf-4e97-bf42-b4c1e13ec70e" (UID: "af92ee67-60cf-4e97-bf42-b4c1e13ec70e"). InnerVolumeSpecName "kube-api-access-tc5k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:25.737255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.737223    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-xks-ca" pod "af92ee67-60cf-4e97-bf42-b4c1e13ec70e" (UID: "af92ee67-60cf-4e97-bf42-b4c1e13ec70e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:25.835476 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.835435    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:25.835660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:25.835487    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5k6\" (UniqueName: \"kubernetes.io/projected/af92ee67-60cf-4e97-bf42-b4c1e13ec70e-kube-api-access-tc5k6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:26.506321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:26.506276    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr" event={"ID":"af92ee67-60cf-4e97-bf42-b4c1e13ec70e","Type":"ContainerDied","Data":"a2dda079505189194ccf984547b5452ce4050a2141337ca8a283943c5c381596"}
Oct 30 20:18:26.506550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:26.506394    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2dda079505189194ccf984547b5452ce4050a2141337ca8a283943c5c381596"
Oct 30 20:18:26.506550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:26.506506    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"
Oct 30 20:18:32.228111 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:32.228068    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:18:32.971232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:32.971164    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp" podUID="8dcf626d-9c08-4db6-b200-b0d17f8b504c" containerName="main" containerID="cri-o://1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6" gracePeriod=45
Oct 30 20:18:33.144288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.144254    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"]
Oct 30 20:18:33.159235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.159009    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d3c508-lf6v2"]
Oct 30 20:18:33.320236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.320211    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:18:33.374156 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.374120    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcf626d-9c08-4db6-b200-b0d17f8b504c-kube-api-access-6xwv2\" (UniqueName: \"kubernetes.io/projected/8dcf626d-9c08-4db6-b200-b0d17f8b504c-kube-api-access-6xwv2\") pod \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") "
Oct 30 20:18:33.374435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.374410    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-home\") pod \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") "
Oct 30 20:18:33.374783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.374760    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-dshm\") pod \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") "
Oct 30 20:18:33.374981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.374960    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-model-cache\") pod \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") "
Oct 30 20:18:33.375139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.375123    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tmp-dir\") pod \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") "
Oct 30 20:18:33.375858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.374810    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-home" pod "8dcf626d-9c08-4db6-b200-b0d17f8b504c" (UID: "8dcf626d-9c08-4db6-b200-b0d17f8b504c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:33.375967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.375702    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-model-cache" pod "8dcf626d-9c08-4db6-b200-b0d17f8b504c" (UID: "8dcf626d-9c08-4db6-b200-b0d17f8b504c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:33.375967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.375813    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tmp-dir" pod "8dcf626d-9c08-4db6-b200-b0d17f8b504c" (UID: "8dcf626d-9c08-4db6-b200-b0d17f8b504c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:33.377169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.377138    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcf626d-9c08-4db6-b200-b0d17f8b504c-kube-api-access-6xwv2" pod "8dcf626d-9c08-4db6-b200-b0d17f8b504c" (UID: "8dcf626d-9c08-4db6-b200-b0d17f8b504c"). InnerVolumeSpecName "kube-api-access-6xwv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:33.380054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.380024    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-dshm" pod "8dcf626d-9c08-4db6-b200-b0d17f8b504c" (UID: "8dcf626d-9c08-4db6-b200-b0d17f8b504c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:33.476585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.476543    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tls-certs\") pod \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\" (UID: \"8dcf626d-9c08-4db6-b200-b0d17f8b504c\") "
Oct 30 20:18:33.476980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.476959    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwv2\" (UniqueName: \"kubernetes.io/projected/8dcf626d-9c08-4db6-b200-b0d17f8b504c-kube-api-access-6xwv2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:33.477081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.476985    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:33.477081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.476998    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:33.477081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.477009    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:33.477081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.477023    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:33.478681 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.478652    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tls-certs" pod "8dcf626d-9c08-4db6-b200-b0d17f8b504c" (UID: "8dcf626d-9c08-4db6-b200-b0d17f8b504c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:33.575108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.575025    4519 generic.go:356] "Generic (PLEG): container finished" podID="8dcf626d-9c08-4db6-b200-b0d17f8b504c" containerID="1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6" exitCode=0
Oct 30 20:18:33.575108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.575075    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"
Oct 30 20:18:33.575108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.575082    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp" event={"ID":"8dcf626d-9c08-4db6-b200-b0d17f8b504c","Type":"ContainerDied","Data":"1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6"}
Oct 30 20:18:33.575358 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.575122    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp" event={"ID":"8dcf626d-9c08-4db6-b200-b0d17f8b504c","Type":"ContainerDied","Data":"2897445b4b167d0825e961fa4d655eeccfdab487b430887b4829ab4f80090330"}
Oct 30 20:18:33.575358 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.575145    4519 scope.go:122] "RemoveContainer" containerID="1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6"
Oct 30 20:18:33.578442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.578405    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf626d-9c08-4db6-b200-b0d17f8b504c-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:33.588211 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.588190    4519 scope.go:122] "RemoveContainer" containerID="1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6"
Oct 30 20:18:33.588547 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:33.588516    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6\": container with ID starting with 1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6 not found: ID does not exist" containerID="1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6"
Oct 30 20:18:33.588672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.588558    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6"} err="failed to get container status \"1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6\": rpc error: code = NotFound desc = could not find container \"1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6\": container with ID starting with 1010b070e3e27dfebe46e6db440867bfcde6aa8f5db0b1d2a1c91206fe6052f6 not found: ID does not exist"
Oct 30 20:18:33.613394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.613372    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4627d467-37c0-4a50-bd4c-d73781f93b24" path="/var/lib/kubelet/pods/4627d467-37c0-4a50-bd4c-d73781f93b24/volumes"
Oct 30 20:18:33.629333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.629310    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"]
Oct 30 20:18:33.633179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:33.633159    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/auth-test-model-27a4d3-kserve-67479f569-tt5zp"]
Oct 30 20:18:35.621775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:35.621743    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dcf626d-9c08-4db6-b200-b0d17f8b504c" path="/var/lib/kubelet/pods/8dcf626d-9c08-4db6-b200-b0d17f8b504c/volumes"
Oct 30 20:18:39.518489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:39.517457    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"]
Oct 30 20:18:39.526496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:39.525562    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64be2-b65cr"]
Oct 30 20:18:39.675497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:39.673739    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af92ee67-60cf-4e97-bf42-b4c1e13ec70e" path="/var/lib/kubelet/pods/af92ee67-60cf-4e97-bf42-b4c1e13ec70e/volumes"
Oct 30 20:18:40.085491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:40.085092    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"]
Oct 30 20:18:40.358548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:40.357876    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd7b75c-64679"]
Oct 30 20:18:41.615535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:41.614106    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cbc1ba5-0593-4017-a8ac-6f4808f4f89d" path="/var/lib/kubelet/pods/6cbc1ba5-0593-4017-a8ac-6f4808f4f89d/volumes"
Oct 30 20:18:41.830953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:41.830913    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"]
Oct 30 20:18:41.831436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:41.831405    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" containerID="cri-o://46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4" gracePeriod=30
Oct 30 20:18:44.255532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:44.255493    4519 scope.go:122] "RemoveContainer" containerID="b714d2e748333fa0c9dfafa169900daec80291e06977ec810c413f939c382ac9"
Oct 30 20:18:44.726489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:44.722802    4519 patch_prober.go:29] interesting pod/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:18:44.726489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:44.722873    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:18:46.033727 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.033650    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"]
Oct 30 20:18:46.045317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.045289    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe359514-qdv88"]
Oct 30 20:18:46.409502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.406370    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"]
Oct 30 20:18:46.409502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.406729    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy" containerID="cri-o://50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842" gracePeriod=30
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.667809    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"]
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.668632    4519 cpu_manager.go:414] "containerMap: removing container" podUID="af92ee67-60cf-4e97-bf42-b4c1e13ec70e" containerName="revoke-keys"
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.668652    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af92ee67-60cf-4e97-bf42-b4c1e13ec70e" containerName="revoke-keys"
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.668679    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8dcf626d-9c08-4db6-b200-b0d17f8b504c" containerName="main"
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.668687    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcf626d-9c08-4db6-b200-b0d17f8b504c" containerName="main"
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.668823    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="af92ee67-60cf-4e97-bf42-b4c1e13ec70e" containerName="revoke-keys"
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.668837    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcf626d-9c08-4db6-b200-b0d17f8b504c" containerName="main"
Oct 30 20:18:46.670499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.668847    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbfc31b-9dd6-42bc-9a7a-f4a5040b2b8e" containerName="istio-proxy"
Oct 30 20:18:46.725558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.723777    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"]
Oct 30 20:18:46.725558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.723892    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.739661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.739172    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"openshift-service-ca.crt\""
Oct 30 20:18:46.739661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.739348    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"kube-root-ca.crt\""
Oct 30 20:18:46.739661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.739536    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"default-dockercfg-v7r9c\""
Oct 30 20:18:46.739661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.739622    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-5c2e32\"/\"sub-test-model-5c2e32-kserve-self-signed-certs\""
Oct 30 20:18:46.841863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.841824    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-model-cache\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.842056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.841928    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tmp-dir\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.842056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.841963    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gshs\" (UniqueName: \"kubernetes.io/projected/1fd518d9-2f66-41af-b22b-dc2d1a7be122-kube-api-access-5gshs\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.842056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.842024    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-dshm\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.842248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.842060    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tls-certs\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.842248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.842091    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-home\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.946633    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-model-cache\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.946732    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tmp-dir\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.946767    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gshs\" (UniqueName: \"kubernetes.io/projected/1fd518d9-2f66-41af-b22b-dc2d1a7be122-kube-api-access-5gshs\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.946832    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-dshm\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.946874    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tls-certs\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.946910    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-home\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.947621    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-home\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.947899    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-model-cache\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.949366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.948164    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tmp-dir\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.969812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.969231    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tls-certs\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:46.986187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:46.986066    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-dshm\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:47.035693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.034208    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gshs\" (UniqueName: \"kubernetes.io/projected/1fd518d9-2f66-41af-b22b-dc2d1a7be122-kube-api-access-5gshs\") pod \"sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") " pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:47.050846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.043625    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:47.148487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.144976    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"]
Oct 30 20:18:47.155959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.155423    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.226103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.218020    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:18:47.228791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.227980    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:47.268736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.262502    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.268736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.262566    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.268736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.262653    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh2zn\" (UniqueName: \"kubernetes.io/projected/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-kube-api-access-fh2zn\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.301719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.301516    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"]
Oct 30 20:18:47.374406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.369607    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.374406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.369707    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh2zn\" (UniqueName: \"kubernetes.io/projected/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-kube-api-access-fh2zn\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.374406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.369976    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.374406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.370772    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.394604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.393947    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh2zn\" (UniqueName: \"kubernetes.io/projected/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-kube-api-access-fh2zn\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.397537 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.397510    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.412202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.412044    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.470791    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-credential-socket\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.470860    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-certs\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.470929    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istiod-ca-cert\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.470977    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-podinfo\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.471039    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-envoy\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.471127    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-data\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.471255    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-kube-api-access-dxjdd\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-kube-api-access-dxjdd\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.471328    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-token\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-token\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.471408    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-socket\") pod \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\" (UID: \"f004e6f6-8245-4a0e-987f-5ef0b5b05604\") "
Oct 30 20:18:47.472616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.472427    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-socket" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:47.475149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.473892    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-credential-socket" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:47.475149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.474307    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istiod-ca-cert" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:47.475149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.474637    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-certs" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:47.475149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.475085    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-data" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:47.479061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.479017    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-kube-api-access-dxjdd" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "kube-api-access-dxjdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:47.479956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.479929    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-envoy" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:47.480189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.480160    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"]
Oct 30 20:18:47.483121 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:18:47.483092    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fd518d9_2f66_41af_b22b_dc2d1a7be122.slice/crio-0b78a5138876cebd2cc11d988127dcdc3d16bcd4109aa929b4b62cb75b6b5e81 WatchSource:0}: Error finding container 0b78a5138876cebd2cc11d988127dcdc3d16bcd4109aa929b4b62cb75b6b5e81: Status 404 returned error can't find the container with id 0b78a5138876cebd2cc11d988127dcdc3d16bcd4109aa929b4b62cb75b6b5e81
Oct 30 20:18:47.483331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.483306    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-token" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:47.488672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.488634    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-podinfo" pod "f004e6f6-8245-4a0e-987f-5ef0b5b05604" (UID: "f004e6f6-8245-4a0e-987f-5ef0b5b05604"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:18:47.573854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573814    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.573854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573851    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.574072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573866    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.574072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573880    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.574072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573892    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.574072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573906    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.574072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573919    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjdd\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-kube-api-access-dxjdd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.574072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573933    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004e6f6-8245-4a0e-987f-5ef0b5b05604-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.574072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.573946    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004e6f6-8245-4a0e-987f-5ef0b5b05604-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:47.592729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.592499    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:47.617036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.617002    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b81735f-d028-4a80-85d4-57cfb011c4b2" path="/var/lib/kubelet/pods/3b81735f-d028-4a80-85d4-57cfb011c4b2/volumes"
Oct 30 20:18:47.821054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.820947    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll" event={"ID":"1fd518d9-2f66-41af-b22b-dc2d1a7be122","Type":"ContainerStarted","Data":"13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459"}
Oct 30 20:18:47.821054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.821006    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll" event={"ID":"1fd518d9-2f66-41af-b22b-dc2d1a7be122","Type":"ContainerStarted","Data":"0b78a5138876cebd2cc11d988127dcdc3d16bcd4109aa929b4b62cb75b6b5e81"}
Oct 30 20:18:47.824173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.824141    4519 generic.go:356] "Generic (PLEG): container finished" podID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerID="46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4" exitCode=0
Oct 30 20:18:47.824334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.824315    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" event={"ID":"f004e6f6-8245-4a0e-987f-5ef0b5b05604","Type":"ContainerDied","Data":"46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4"}
Oct 30 20:18:47.824389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.824368    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl" event={"ID":"f004e6f6-8245-4a0e-987f-5ef0b5b05604","Type":"ContainerDied","Data":"d2a9aae6cd2a05433c08c9dea001d8d95307099b4cd367051d02a26cd8ebf5f8"}
Oct 30 20:18:47.824423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.824401    4519 scope.go:122] "RemoveContainer" containerID="46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4"
Oct 30 20:18:47.824621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.824595    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"
Oct 30 20:18:47.836880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.836859    4519 scope.go:122] "RemoveContainer" containerID="02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9"
Oct 30 20:18:47.848080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.848051    4519 scope.go:122] "RemoveContainer" containerID="46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4"
Oct 30 20:18:47.848372 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:47.848351    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4\": container with ID starting with 46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4 not found: ID does not exist" containerID="46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4"
Oct 30 20:18:47.848514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.848385    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4"} err="failed to get container status \"46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4\": rpc error: code = NotFound desc = could not find container \"46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4\": container with ID starting with 46cca5fed7bdc826e596b47dce5d813076fdc35965afe947fd38355d15ffe5d4 not found: ID does not exist"
Oct 30 20:18:47.848514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.848403    4519 scope.go:122] "RemoveContainer" containerID="02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9"
Oct 30 20:18:47.848698 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:47.848671    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9\": container with ID starting with 02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9 not found: ID does not exist" containerID="02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9"
Oct 30 20:18:47.848777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.848708    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9"} err="failed to get container status \"02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9\": rpc error: code = NotFound desc = could not find container \"02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9\": container with ID starting with 02084fa0e39fb966b3247bbb16c419c40c3548007bc85d5ffcc592550f720dc9 not found: ID does not exist"
Oct 30 20:18:47.883919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.883857    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll" podStartSLOduration=1.88383697 podStartE2EDuration="1.88383697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:47.865872606 +0000 UTC m=+2652.366576331" watchObservedRunningTime="2026-10-30 20:18:47.88383697 +0000 UTC m=+2652.384540695"
Oct 30 20:18:47.912337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.912152    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"]
Oct 30 20:18:47.920127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.920096    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d3c508-openshift-default-598bd9687-jpgtl"]
Oct 30 20:18:47.927954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:47.927925    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"]
Oct 30 20:18:47.928455 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:18:47.928425    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fd65b7_00ee_4a3e_969b_ca8a150dc735.slice/crio-ca4a0aea5057b4d6fc818dc60c81e4392da814cae997f6924fb4e828cc08ba9b WatchSource:0}: Error finding container ca4a0aea5057b4d6fc818dc60c81e4392da814cae997f6924fb4e828cc08ba9b: Status 404 returned error can't find the container with id ca4a0aea5057b4d6fc818dc60c81e4392da814cae997f6924fb4e828cc08ba9b
Oct 30 20:18:48.033100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:48.033032    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"]
Oct 30 20:18:48.039108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:48.039077    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addf6c77-24zqs"]
Oct 30 20:18:48.046150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:48.046124    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:48.834011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:48.833805    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf" event={"ID":"a9fd65b7-00ee-4a3e-969b-ca8a150dc735","Type":"ContainerStarted","Data":"bfca253ec0e8974d341a547333739ec6e0669e07b9003aecfc29224ba92f8b86"}
Oct 30 20:18:48.834011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:48.833873    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf" event={"ID":"a9fd65b7-00ee-4a3e-969b-ca8a150dc735","Type":"ContainerStarted","Data":"ca4a0aea5057b4d6fc818dc60c81e4392da814cae997f6924fb4e828cc08ba9b"}
Oct 30 20:18:48.882574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:48.882511    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf" podStartSLOduration=1.882490553 podStartE2EDuration="1.882490553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:48.877240186 +0000 UTC m=+2653.377943911" watchObservedRunningTime="2026-10-30 20:18:48.882490553 +0000 UTC m=+2653.383194273"
Oct 30 20:18:49.615166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:49.615075    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68dc8b8a-693f-436e-9219-f91abd6b1b94" path="/var/lib/kubelet/pods/68dc8b8a-693f-436e-9219-f91abd6b1b94/volumes"
Oct 30 20:18:49.615736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:49.615580    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" path="/var/lib/kubelet/pods/f004e6f6-8245-4a0e-987f-5ef0b5b05604/volumes"
Oct 30 20:18:49.848019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:49.847974    4519 generic.go:356] "Generic (PLEG): container finished" podID="a9fd65b7-00ee-4a3e-969b-ca8a150dc735" containerID="bfca253ec0e8974d341a547333739ec6e0669e07b9003aecfc29224ba92f8b86" exitCode=0
Oct 30 20:18:49.848436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:49.848052    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf" event={"ID":"a9fd65b7-00ee-4a3e-969b-ca8a150dc735","Type":"ContainerDied","Data":"bfca253ec0e8974d341a547333739ec6e0669e07b9003aecfc29224ba92f8b86"}
Oct 30 20:18:50.861221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:50.861141    4519 generic.go:356] "Generic (PLEG): container finished" podID="85027075-443e-4323-a372-86b86a6dd098" containerID="b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451" exitCode=0
Oct 30 20:18:50.861782 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:50.861220    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" event={"ID":"85027075-443e-4323-a372-86b86a6dd098","Type":"ContainerDied","Data":"b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451"}
Oct 30 20:18:50.861855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:50.861822    4519 scope.go:122] "RemoveContainer" containerID="b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451"
Oct 30 20:18:50.864102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:50.864067    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:18:50.864221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:50.864162    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:18:50.864281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:50.864233    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:18:51.305045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.305020    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:51.478696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.478619    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-xks-ca\") pod \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") "
Oct 30 20:18:51.478943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.478922    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-openshift-ca\") pod \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") "
Oct 30 20:18:51.479017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.479002    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-kube-api-access-fh2zn\" (UniqueName: \"kubernetes.io/projected/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-kube-api-access-fh2zn\") pod \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\" (UID: \"a9fd65b7-00ee-4a3e-969b-ca8a150dc735\") "
Oct 30 20:18:51.479414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.479371    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-openshift-ca" pod "a9fd65b7-00ee-4a3e-969b-ca8a150dc735" (UID: "a9fd65b7-00ee-4a3e-969b-ca8a150dc735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:51.481556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.481525    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-kube-api-access-fh2zn" pod "a9fd65b7-00ee-4a3e-969b-ca8a150dc735" (UID: "a9fd65b7-00ee-4a3e-969b-ca8a150dc735"). InnerVolumeSpecName "kube-api-access-fh2zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:51.481670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.481545    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-xks-ca" pod "a9fd65b7-00ee-4a3e-969b-ca8a150dc735" (UID: "a9fd65b7-00ee-4a3e-969b-ca8a150dc735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:51.581968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.581932    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.581968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.581969    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh2zn\" (UniqueName: \"kubernetes.io/projected/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-kube-api-access-fh2zn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.582209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.581986    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fd65b7-00ee-4a3e-969b-ca8a150dc735-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.682363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.682335    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:18:51.784576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.784492    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-socket\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.784729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.784591    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-kube-api-access-w7chv\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-kube-api-access-w7chv\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.784729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.784627    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/726bac57-6740-4469-bb0e-5a073dac1490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726bac57-6740-4469-bb0e-5a073dac1490-istio-podinfo\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.784826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.784740    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726bac57-6740-4469-bb0e-5a073dac1490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726bac57-6740-4469-bb0e-5a073dac1490-istiod-ca-cert\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.784865 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.784811    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-socket" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.784865 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.784821    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-istio-token\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-istio-token\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.784968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.784947    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-credential-socket\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.785028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785014    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-certs\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.785071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785048    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-envoy\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.785117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785087    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-data\") pod \"726bac57-6740-4469-bb0e-5a073dac1490\" (UID: \"726bac57-6740-4469-bb0e-5a073dac1490\") "
Oct 30 20:18:51.785796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785549    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726bac57-6740-4469-bb0e-5a073dac1490-istiod-ca-cert" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:51.785796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785762    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-data" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.786018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785915    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-credential-socket" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.786018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785954    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726bac57-6740-4469-bb0e-5a073dac1490-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.786018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785975    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.786018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785976    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-certs" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.786018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.785988    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.788086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.788056    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/726bac57-6740-4469-bb0e-5a073dac1490-istio-podinfo" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:18:51.788971 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.788943    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-kube-api-access-w7chv" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "kube-api-access-w7chv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:51.789081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.788966    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-envoy" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.789081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.789074    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-istio-token" pod "726bac57-6740-4469-bb0e-5a073dac1490" (UID: "726bac57-6740-4469-bb0e-5a073dac1490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:51.876127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.876080    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" event={"ID":"85027075-443e-4323-a372-86b86a6dd098","Type":"ContainerStarted","Data":"acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1"}
Oct 30 20:18:51.878587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.878544    4519 generic.go:356] "Generic (PLEG): container finished" podID="726bac57-6740-4469-bb0e-5a073dac1490" containerID="50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842" exitCode=0
Oct 30 20:18:51.878742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.878729    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" event={"ID":"726bac57-6740-4469-bb0e-5a073dac1490","Type":"ContainerDied","Data":"50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842"}
Oct 30 20:18:51.878813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.878771    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc" event={"ID":"726bac57-6740-4469-bb0e-5a073dac1490","Type":"ContainerDied","Data":"b872d057014f52ed3da85c25371c04b07172b7881416986c5327eeefe50a111b"}
Oct 30 20:18:51.878813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.878800    4519 scope.go:122] "RemoveContainer" containerID="50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842"
Oct 30 20:18:51.878941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.878928    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"
Oct 30 20:18:51.887051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887022    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.887051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887052    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.887284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887067    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.887284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887080    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726bac57-6740-4469-bb0e-5a073dac1490-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.887284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887094    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7chv\" (UniqueName: \"kubernetes.io/projected/726bac57-6740-4469-bb0e-5a073dac1490-kube-api-access-w7chv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.887284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887106    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726bac57-6740-4469-bb0e-5a073dac1490-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:18:51.887596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887290    4519 generic.go:356] "Generic (PLEG): container finished" podID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerID="b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982" exitCode=0
Oct 30 20:18:51.887596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887420    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" event={"ID":"c6a08deb-6080-41c4-baea-37afb9d69f36","Type":"ContainerDied","Data":"b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982"}
Oct 30 20:18:51.887853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.887808    4519 scope.go:122] "RemoveContainer" containerID="b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982"
Oct 30 20:18:51.889904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.889823    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:18:51.890035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.889910    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:18:51.890035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.889960    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:18:51.892573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.892503    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf" event={"ID":"a9fd65b7-00ee-4a3e-969b-ca8a150dc735","Type":"ContainerDied","Data":"ca4a0aea5057b4d6fc818dc60c81e4392da814cae997f6924fb4e828cc08ba9b"}
Oct 30 20:18:51.892573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.892550    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca4a0aea5057b4d6fc818dc60c81e4392da814cae997f6924fb4e828cc08ba9b"
Oct 30 20:18:51.892746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.892607    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"
Oct 30 20:18:51.901266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.901245    4519 scope.go:122] "RemoveContainer" containerID="4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e"
Oct 30 20:18:51.914247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.914222    4519 scope.go:122] "RemoveContainer" containerID="50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842"
Oct 30 20:18:51.914702 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:51.914641    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842\": container with ID starting with 50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842 not found: ID does not exist" containerID="50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842"
Oct 30 20:18:51.914702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.914675    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842"} err="failed to get container status \"50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842\": rpc error: code = NotFound desc = could not find container \"50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842\": container with ID starting with 50db6c3a38e272305b49095d6a2427cd6c4488abcff839c3b288e6c3072d1842 not found: ID does not exist"
Oct 30 20:18:51.914702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.914701    4519 scope.go:122] "RemoveContainer" containerID="4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e"
Oct 30 20:18:51.914943 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:18:51.914922    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e\": container with ID starting with 4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e not found: ID does not exist" containerID="4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e"
Oct 30 20:18:51.915001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:51.914950    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e"} err="failed to get container status \"4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e\": rpc error: code = NotFound desc = could not find container \"4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e\": container with ID starting with 4768469084c0523db1484f5e9096259fd7a926fd18bc79b0a1f3dc6f6e328d1e not found: ID does not exist"
Oct 30 20:18:52.029694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.029660    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"]
Oct 30 20:18:52.035991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.035917    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-6cfbc"]
Oct 30 20:18:52.040140 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.040113    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"]
Oct 30 20:18:52.043522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.043497    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8945b2f7-g7pj7"]
Oct 30 20:18:52.840203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.840162    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:18:52.841619 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.841596    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 20:18:52.841687 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.841644    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 20:18:52.908323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:52.908283    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" event={"ID":"c6a08deb-6080-41c4-baea-37afb9d69f36","Type":"ContainerStarted","Data":"9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c"}
Oct 30 20:18:53.171373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.171333    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"]
Oct 30 20:18:53.172532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.172501    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a9fd65b7-00ee-4a3e-969b-ca8a150dc735" containerName="revoke-keys"
Oct 30 20:18:53.172838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.172820    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fd65b7-00ee-4a3e-969b-ca8a150dc735" containerName="revoke-keys"
Oct 30 20:18:53.172987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.172975    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy"
Oct 30 20:18:53.173089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173077    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy"
Oct 30 20:18:53.173191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173181    4519 cpu_manager.go:414] "containerMap: removing container" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy"
Oct 30 20:18:53.173272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173261    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy"
Oct 30 20:18:53.173379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173368    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy"
Oct 30 20:18:53.173481 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173451    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy"
Oct 30 20:18:53.173591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173579    4519 cpu_manager.go:414] "containerMap: removing container" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy"
Oct 30 20:18:53.173678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173667    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy"
Oct 30 20:18:53.173937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.173921    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy"
Oct 30 20:18:53.174042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.174030    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy"
Oct 30 20:18:53.174147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.174136    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004e6f6-8245-4a0e-987f-5ef0b5b05604" containerName="istio-proxy"
Oct 30 20:18:53.174244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.174234    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fd65b7-00ee-4a3e-969b-ca8a150dc735" containerName="revoke-keys"
Oct 30 20:18:53.191582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.191551    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.194514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.194483    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"]
Oct 30 20:18:53.206859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.206826    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-043892\"/\"default-dockercfg-826k4\""
Oct 30 20:18:53.207140 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.207123    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-043892\"/\"sub-test-model-043892-kserve-self-signed-certs\""
Oct 30 20:18:53.208007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.207894    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-043892\"/\"kube-root-ca.crt\""
Oct 30 20:18:53.208096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.207946    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-043892\"/\"openshift-service-ca.crt\""
Oct 30 20:18:53.228264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.228232    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-dshm\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.228443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.228298    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tmp-dir\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.228443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.228428    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5sqd\" (UniqueName: \"kubernetes.io/projected/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-kube-api-access-n5sqd\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.228600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.228557    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tls-certs\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.228669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.228658    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-home\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.228730 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.228689    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-model-cache\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.336008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.335957    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-home\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.336193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.336035    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-model-cache\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.336193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.336126    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-dshm\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.336193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.336189    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tmp-dir\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.336396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.336336    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5sqd\" (UniqueName: \"kubernetes.io/projected/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-kube-api-access-n5sqd\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.336485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.336421    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tls-certs\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.338791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.338759    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-home\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.339096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.339072    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-model-cache\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.344531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.339413    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tmp-dir\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.345783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.345763    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tls-certs\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.363167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.361768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-dshm\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.382802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.382732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5sqd\" (UniqueName: \"kubernetes.io/projected/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-kube-api-access-n5sqd\") pod \"sub-test-model-043892-kserve-7895c6cb6f-dscvj\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") " pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.519030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.518924    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:53.724527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.713573    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726bac57-6740-4469-bb0e-5a073dac1490" path="/var/lib/kubelet/pods/726bac57-6740-4469-bb0e-5a073dac1490/volumes"
Oct 30 20:18:53.725613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.725579    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84947a8f-f3cc-43b2-a5cc-612bfda89855" path="/var/lib/kubelet/pods/84947a8f-f3cc-43b2-a5cc-612bfda89855/volumes"
Oct 30 20:18:53.787998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.787906    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:18:53.788354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.788326    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 20:18:53.788549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.788523    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 20:18:53.841499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.840951    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 20:18:53.841499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:53.841019    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 20:18:54.335570 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:18:54.335541    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde7828ce_aeb7_4ab5_9a96_2dfbc685f37d.slice/crio-76f5c45c3b04634193ad7b5e50c31193a2f669ab9f00c600e1f175f67817c8f5 WatchSource:0}: Error finding container 76f5c45c3b04634193ad7b5e50c31193a2f669ab9f00c600e1f175f67817c8f5: Status 404 returned error can't find the container with id 76f5c45c3b04634193ad7b5e50c31193a2f669ab9f00c600e1f175f67817c8f5
Oct 30 20:18:54.337270 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:54.337246    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"]
Oct 30 20:18:54.742723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:54.742688    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 20:18:54.742895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:54.742757    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 20:18:54.840083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:54.840047    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 20:18:54.840240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:54.840098    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 20:18:55.000543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:54.997873    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj" event={"ID":"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d","Type":"ContainerStarted","Data":"9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d"}
Oct 30 20:18:55.000543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:54.997933    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj" event={"ID":"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d","Type":"ContainerStarted","Data":"76f5c45c3b04634193ad7b5e50c31193a2f669ab9f00c600e1f175f67817c8f5"}
Oct 30 20:18:55.040976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:55.040794    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj" podStartSLOduration=2.040778989 podStartE2EDuration="2.040778989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:55.040129839 +0000 UTC m=+2659.540833565" watchObservedRunningTime="2026-10-30 20:18:55.040778989 +0000 UTC m=+2659.541482713"
Oct 30 20:18:55.519780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:55.519741    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:18:55.597497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:55.593581    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:18:55.743758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:55.743719    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 20:18:55.743974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:55.743770    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 20:18:55.839948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:55.839865    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 20:18:55.839948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:55.839918    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 20:18:56.743230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:56.743195    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 20:18:56.743726 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:56.743253    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 20:18:56.839830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:56.839790    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 20:18:56.840031 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:56.839845    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 20:18:57.069541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:57.061984    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:57.132531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:57.131255    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:57.743087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:57.743055    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 20:18:57.743319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:57.743299    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 20:18:57.840768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:57.840172    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 20:18:57.840768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:57.840255    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 20:18:58.101546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:58.094906    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:18:58.743068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:58.743026    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 20:18:58.743295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:58.743093    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 20:18:58.940270 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:58.940216    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:18:58.941509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:58.941454    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:18:59.731103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:59.731067    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:18:59.857384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:18:59.857352    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:19:00.056387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:00.056302    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:19:02.221985 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:19:02.221940    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:19:03.523188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:03.522900    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:19:03.539876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:03.539846    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:19:04.111530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:04.111498    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:19:06.032487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:06.032438    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"]
Oct 30 20:19:06.044711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:06.044679    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1583ef96-4758m"]
Oct 30 20:19:07.028933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:07.028901    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"]
Oct 30 20:19:07.040573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:07.040545    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47f32db-ltmt5"]
Oct 30 20:19:07.613057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:07.613025    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008571ad-6703-4b26-9ff7-6cdc8533932a" path="/var/lib/kubelet/pods/008571ad-6703-4b26-9ff7-6cdc8533932a/volumes"
Oct 30 20:19:07.613529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:07.613513    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55399c07-fbfe-474d-af29-e9c06ef80592" path="/var/lib/kubelet/pods/55399c07-fbfe-474d-af29-e9c06ef80592/volumes"
Oct 30 20:19:28.053787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:28.053750    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"]
Oct 30 20:19:28.058779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:28.058749    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cfcee-k8kwf"]
Oct 30 20:19:29.613490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:29.613439    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fd65b7-00ee-4a3e-969b-ca8a150dc735" path="/var/lib/kubelet/pods/a9fd65b7-00ee-4a3e-969b-ca8a150dc735/volumes"
Oct 30 20:19:32.362549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.362511    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"]
Oct 30 20:19:32.364093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.364069    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="726bac57-6740-4469-bb0e-5a073dac1490" containerName="istio-proxy"
Oct 30 20:19:32.370494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.370049    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:32.394723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.394692    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"]
Oct 30 20:19:32.402543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.402518    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-188d45-dockercfg-k948f\""
Oct 30 20:19:32.402826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.402811    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-188d45\""
Oct 30 20:19:32.417337 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:19:32.417308    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:19:32.491855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.491824    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:32.492115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.492101    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnrn\" (UniqueName: \"kubernetes.io/projected/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-kube-api-access-6hnrn\") pod \"payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:32.593352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.593307    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnrn\" (UniqueName: \"kubernetes.io/projected/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-kube-api-access-6hnrn\") pod \"payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:32.593651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.593626    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:32.594658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.594630    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:32.613508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.613413    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnrn\" (UniqueName: \"kubernetes.io/projected/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-kube-api-access-6hnrn\") pod \"payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:32.695850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:32.695808    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:33.086537 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:33.083789    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"]
Oct 30 20:19:33.086537 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:19:33.083859    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9bf1a0_d8ce_4c79_b412_8c9852cd6e9e.slice/crio-6a0d30306f37a44589ce8e5a902b2a1c4071662e78e4086e6e45df1b4096d51a WatchSource:0}: Error finding container 6a0d30306f37a44589ce8e5a902b2a1c4071662e78e4086e6e45df1b4096d51a: Status 404 returned error can't find the container with id 6a0d30306f37a44589ce8e5a902b2a1c4071662e78e4086e6e45df1b4096d51a
Oct 30 20:19:33.090060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:33.090040    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:19:33.417541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:33.417487    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p" event={"ID":"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e","Type":"ContainerStarted","Data":"6a0d30306f37a44589ce8e5a902b2a1c4071662e78e4086e6e45df1b4096d51a"}
Oct 30 20:19:36.461489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:36.461260    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p" event={"ID":"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e","Type":"ContainerStarted","Data":"77f63be8be6d2a8435a76504bb22fddd764a2cf318b32a1c0db0bfca3d3fa028"}
Oct 30 20:19:36.462003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:36.461630    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:36.510486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:36.510292    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p" podStartSLOduration=1.9374533280000001 podStartE2EDuration="4.5102747s" totalImagesPullingTime="2.572821372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:36.509715316 +0000 UTC m=+2701.010419042" watchObservedRunningTime="2026-10-30 20:19:36.5102747 +0000 UTC m=+2701.010978425"
Oct 30 20:19:42.594028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:42.593978    4519 generic.go:356] "Generic (PLEG): container finished" podID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerID="3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46" exitCode=0
Oct 30 20:19:42.594591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:42.594058    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" event={"ID":"1781aa1d-20cb-44a8-b7f8-0004676efc46","Type":"ContainerDied","Data":"3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46"}
Oct 30 20:19:42.594678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:42.594648    4519 scope.go:122] "RemoveContainer" containerID="3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46"
Oct 30 20:19:42.596686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:42.596654    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:19:42.596832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:42.596742    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:19:42.596832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:42.596788    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:19:42.905630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:42.905536    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:19:43.604582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:43.604537    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" event={"ID":"1781aa1d-20cb-44a8-b7f8-0004676efc46","Type":"ContainerStarted","Data":"5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64"}
Oct 30 20:19:44.360497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:44.357646    4519 scope.go:122] "RemoveContainer" containerID="f1ddf8eac8f1ea99cac9fe7a4158bc0c6d439453bdd7e87b3851d46dfac1a17c"
Oct 30 20:19:44.378494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:44.375069    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:19:44.378494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:44.375225    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 20:19:44.378494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:44.375301    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 20:19:44.387265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:44.387244    4519 scope.go:122] "RemoveContainer" containerID="f71bac934edb82a80b63691bc6856052fd0620dadfcbca2b170467d961bf71dc"
Oct 30 20:19:44.400009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:44.399979    4519 scope.go:122] "RemoveContainer" containerID="01ca2b70b0e20f426cfc5a9b7ca4b8e2749eca0e61bcec57dfd5b5f7efc71686"
Oct 30 20:19:44.417704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:44.417672    4519 scope.go:122] "RemoveContainer" containerID="94f9c4bf6dc3b45f36bd189381aaf94e0a62bc00193007b3a8ba13406de6f72b"
Oct 30 20:19:45.377943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:45.377902    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 20:19:45.378530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:45.377977    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 20:19:46.375939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:46.375895    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 20:19:46.376221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:46.376186    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 20:19:47.378492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:47.374440    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 20:19:47.378492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:47.374549    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 20:19:47.526615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:47.526582    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:19:48.378940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:48.378868    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 20:19:48.379651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:48.379614    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 20:19:49.380226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:49.380191    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:19:49.380959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:49.380925    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:19:49.381723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:49.381699    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:19:52.010065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.010021    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"]
Oct 30 20:19:52.016281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.016252    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.043579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.043546    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-043892-openshift-default-dockercfg-hvz52\""
Oct 30 20:19:52.060297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.060255    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"]
Oct 30 20:19:52.130717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.130676    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.130717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.130727    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-kube-api-access-km85r\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.131002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.130833    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.131002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.130871    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.131002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.130894    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.131175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.130997    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.131175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.131108    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.131289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.131180    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.131289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.131256    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.232885 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.232849    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.232899    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-kube-api-access-km85r\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.232952    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.232991    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.233020    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.233048    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.233130    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.233184    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.233394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.233240    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.235216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.235179    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.235507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.235447    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.238588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.236550    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.239048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.239014    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.239581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.239559    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.239994 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.239968    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.241061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.241034    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.255660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.255630    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.256055 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.256027    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-kube-api-access-km85r\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.330720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.330616    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:52.675275 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.675236    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"]
Oct 30 20:19:52.677626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.677579    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:19:52.677766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.677677    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:19:52.677766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:52.677725    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:19:53.695571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:53.695525    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" event={"ID":"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f","Type":"ContainerStarted","Data":"7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b"}
Oct 30 20:19:53.696096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:53.695598    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" event={"ID":"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f","Type":"ContainerStarted","Data":"56b83b10f1177a6752bd3e950c5e3ea20d4d51f8ef84411e5a85c7ce35d402ca"}
Oct 30 20:19:53.753221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:53.753147    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podStartSLOduration=2.753127651 podStartE2EDuration="2.753127651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:53.749809061 +0000 UTC m=+2718.250512788" watchObservedRunningTime="2026-10-30 20:19:53.753127651 +0000 UTC m=+2718.253831376"
Oct 30 20:19:54.331725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:54.331684    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:19:54.332840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:54.332812    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 20:19:54.333003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:54.332853    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 20:19:55.333498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:55.332731    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 20:19:55.333498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:55.332779    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 20:19:56.331817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:56.331782    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 20:19:56.332008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:56.331837    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 20:19:57.331546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.331504    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 20:19:57.331979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.331564    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 20:19:57.394011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.393967    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"]
Oct 30 20:19:57.417491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.413413    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.484864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.479709    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-model-cache\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.484864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.479773    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-dshm\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.484864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.479803    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tls-certs\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.484864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.479852    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-home\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.484864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.479884    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nxn\" (UniqueName: \"kubernetes.io/projected/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-kube-api-access-d8nxn\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.484864 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.479948    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tmp-dir\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.524580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.514518    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"openshift-service-ca.crt\""
Oct 30 20:19:57.524580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.518531    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"kube-root-ca.crt\""
Oct 30 20:19:57.532490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.531092    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"default-dockercfg-2spx4\""
Oct 30 20:19:57.582554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.556670    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-27a4d3\"/\"sub-test-model-27a4d3-kserve-self-signed-certs\""
Oct 30 20:19:57.633489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.628523    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tmp-dir\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.633489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.628812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-model-cache\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.633489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.628855    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-dshm\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.633489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.628875    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tls-certs\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.633489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.628917    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-home\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.633489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.628954    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nxn\" (UniqueName: \"kubernetes.io/projected/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-kube-api-access-d8nxn\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.690588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.655880    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-home\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.690588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.683370    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tmp-dir\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.690588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.683650    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-model-cache\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.846542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.825804    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-dshm\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.953645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.952019    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nxn\" (UniqueName: \"kubernetes.io/projected/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-kube-api-access-d8nxn\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.953645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.952604    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tls-certs\") pod \"sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") " pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:57.957624 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.955353    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"]
Oct 30 20:19:57.994392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:57.994081    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:19:58.331682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:58.331638    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 20:19:58.332152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:58.332123    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 20:19:58.489703 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:19:58.489665    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9ae9196_8b5a_4d86_840f_67e3d79a0cad.slice/crio-2306733f7fc329614bfe68850d953e805f1c18bb602df75513374381dbd23438 WatchSource:0}: Error finding container 2306733f7fc329614bfe68850d953e805f1c18bb602df75513374381dbd23438: Status 404 returned error can't find the container with id 2306733f7fc329614bfe68850d953e805f1c18bb602df75513374381dbd23438
Oct 30 20:19:58.491617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:58.491582    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"]
Oct 30 20:19:59.014312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:59.014269    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4" event={"ID":"e9ae9196-8b5a-4d86-840f-67e3d79a0cad","Type":"ContainerStarted","Data":"339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98"}
Oct 30 20:19:59.014544 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:59.014322    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4" event={"ID":"e9ae9196-8b5a-4d86-840f-67e3d79a0cad","Type":"ContainerStarted","Data":"2306733f7fc329614bfe68850d953e805f1c18bb602df75513374381dbd23438"}
Oct 30 20:19:59.077695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:59.077630    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4" podStartSLOduration=2.077607329 podStartE2EDuration="2.077607329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:59.057725109 +0000 UTC m=+2723.558428836" watchObservedRunningTime="2026-10-30 20:19:59.077607329 +0000 UTC m=+2723.578311054"
Oct 30 20:19:59.331991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:59.331899    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 20:19:59.331991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:59.331968    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 20:19:59.999229 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:19:59.999185    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:20:00.346257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:00.346180    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:20:00.346880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:00.346855    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:20:00.347957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:00.347934    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:20:02.228829 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:20:02.228783    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:20:07.995991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:07.995952    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:20:08.009150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:08.009116    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:20:08.218664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:08.218628    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:20:12.009875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.009833    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"]
Oct 30 20:20:12.014198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.014178    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.028435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.028407    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:12.040453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.040426    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:20:12.045000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.044964    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"]
Oct 30 20:20:12.163639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.163600    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.163835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.163685    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd95z\" (UniqueName: \"kubernetes.io/projected/0d136ce1-3be3-44ed-a735-0c02b77ccc05-kube-api-access-bd95z\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.163835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.163768    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.265339 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.265243    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.265533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.265342    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd95z\" (UniqueName: \"kubernetes.io/projected/0d136ce1-3be3-44ed-a735-0c02b77ccc05-kube-api-access-bd95z\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.265533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.265415    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.266183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.266154    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.267787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.267757    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.284991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.284965    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd95z\" (UniqueName: \"kubernetes.io/projected/0d136ce1-3be3-44ed-a735-0c02b77ccc05-kube-api-access-bd95z\") pod \"maas-api-revoke-sub-673dc401-w2zkj\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.329203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.329160    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:12.642262 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:20:12.642217    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d136ce1_3be3_44ed_a735_0c02b77ccc05.slice/crio-35760b0416fbcfba575fef449a6c91ba8cfa7aa1c138b896f6c7890342e262c0 WatchSource:0}: Error finding container 35760b0416fbcfba575fef449a6c91ba8cfa7aa1c138b896f6c7890342e262c0: Status 404 returned error can't find the container with id 35760b0416fbcfba575fef449a6c91ba8cfa7aa1c138b896f6c7890342e262c0
Oct 30 20:20:12.643461 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:12.643433    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"]
Oct 30 20:20:13.251649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:13.251540    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj" event={"ID":"0d136ce1-3be3-44ed-a735-0c02b77ccc05","Type":"ContainerStarted","Data":"80af9f9d7add2184c59e30ec5fb08da702ccd6ce664cb6e80cc01057fd657fff"}
Oct 30 20:20:13.251649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:13.251610    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj" event={"ID":"0d136ce1-3be3-44ed-a735-0c02b77ccc05","Type":"ContainerStarted","Data":"35760b0416fbcfba575fef449a6c91ba8cfa7aa1c138b896f6c7890342e262c0"}
Oct 30 20:20:13.291593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:13.291534    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj" podStartSLOduration=2.291519504 podStartE2EDuration="2.291519504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:13.290299735 +0000 UTC m=+2737.791003476" watchObservedRunningTime="2026-10-30 20:20:13.291519504 +0000 UTC m=+2737.792223227"
Oct 30 20:20:14.262778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:14.262733    4519 generic.go:356] "Generic (PLEG): container finished" podID="0d136ce1-3be3-44ed-a735-0c02b77ccc05" containerID="80af9f9d7add2184c59e30ec5fb08da702ccd6ce664cb6e80cc01057fd657fff" exitCode=0
Oct 30 20:20:14.263225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:14.262820    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj" event={"ID":"0d136ce1-3be3-44ed-a735-0c02b77ccc05","Type":"ContainerDied","Data":"80af9f9d7add2184c59e30ec5fb08da702ccd6ce664cb6e80cc01057fd657fff"}
Oct 30 20:20:15.443591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.443545    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:15.520491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.513993    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-xks-ca\") pod \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") "
Oct 30 20:20:15.520491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.514090    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-openshift-ca\") pod \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") "
Oct 30 20:20:15.520491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.514190    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d136ce1-3be3-44ed-a735-0c02b77ccc05-kube-api-access-bd95z\" (UniqueName: \"kubernetes.io/projected/0d136ce1-3be3-44ed-a735-0c02b77ccc05-kube-api-access-bd95z\") pod \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\" (UID: \"0d136ce1-3be3-44ed-a735-0c02b77ccc05\") "
Oct 30 20:20:15.536531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.526990    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-openshift-ca" pod "0d136ce1-3be3-44ed-a735-0c02b77ccc05" (UID: "0d136ce1-3be3-44ed-a735-0c02b77ccc05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:15.543454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.543418    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d136ce1-3be3-44ed-a735-0c02b77ccc05-kube-api-access-bd95z" pod "0d136ce1-3be3-44ed-a735-0c02b77ccc05" (UID: "0d136ce1-3be3-44ed-a735-0c02b77ccc05"). InnerVolumeSpecName "kube-api-access-bd95z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:15.548333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.548299    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-xks-ca" pod "0d136ce1-3be3-44ed-a735-0c02b77ccc05" (UID: "0d136ce1-3be3-44ed-a735-0c02b77ccc05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:15.625244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.625205    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:20:15.625244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.625240    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d136ce1-3be3-44ed-a735-0c02b77ccc05-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:20:15.625527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:15.625256    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd95z\" (UniqueName: \"kubernetes.io/projected/0d136ce1-3be3-44ed-a735-0c02b77ccc05-kube-api-access-bd95z\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:20:16.284446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:16.284414    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"
Oct 30 20:20:16.284657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:16.284450    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj" event={"ID":"0d136ce1-3be3-44ed-a735-0c02b77ccc05","Type":"ContainerDied","Data":"35760b0416fbcfba575fef449a6c91ba8cfa7aa1c138b896f6c7890342e262c0"}
Oct 30 20:20:16.284657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:16.284526    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35760b0416fbcfba575fef449a6c91ba8cfa7aa1c138b896f6c7890342e262c0"
Oct 30 20:20:22.518700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.518606    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"]
Oct 30 20:20:22.519199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.519168    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0d136ce1-3be3-44ed-a735-0c02b77ccc05" containerName="revoke-keys"
Oct 30 20:20:22.519199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.519180    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d136ce1-3be3-44ed-a735-0c02b77ccc05" containerName="revoke-keys"
Oct 30 20:20:22.519328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.519284    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d136ce1-3be3-44ed-a735-0c02b77ccc05" containerName="revoke-keys"
Oct 30 20:20:22.522910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.522882    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.554314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.554282    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"]
Oct 30 20:20:22.560760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.560732    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:22.560967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.560802    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:20:22.650305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.650263    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.650499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.650336    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.650499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.650436    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9cp\" (UniqueName: \"kubernetes.io/projected/8e28ee8c-8795-4313-8b01-92845e6c40e3-kube-api-access-6x9cp\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.752679 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.752639    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.752878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.752714    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.752878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.752770    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9cp\" (UniqueName: \"kubernetes.io/projected/8e28ee8c-8795-4313-8b01-92845e6c40e3-kube-api-access-6x9cp\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.753482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.753440    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.755767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.755740    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.772694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.772635    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9cp\" (UniqueName: \"kubernetes.io/projected/8e28ee8c-8795-4313-8b01-92845e6c40e3-kube-api-access-6x9cp\") pod \"maas-api-revoke-sub-6d0b9bc1-xhcsb\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:22.837580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:22.837551    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:23.155815 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:20:23.153898    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e28ee8c_8795_4313_8b01_92845e6c40e3.slice/crio-9500fd51e670fb2eec46b3cc96a656fdf45abbfce4d5bb74d137f541e8fda6b3 WatchSource:0}: Error finding container 9500fd51e670fb2eec46b3cc96a656fdf45abbfce4d5bb74d137f541e8fda6b3: Status 404 returned error can't find the container with id 9500fd51e670fb2eec46b3cc96a656fdf45abbfce4d5bb74d137f541e8fda6b3
Oct 30 20:20:23.155815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:23.154327    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"]
Oct 30 20:20:23.347387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:23.347352    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb" event={"ID":"8e28ee8c-8795-4313-8b01-92845e6c40e3","Type":"ContainerStarted","Data":"9500fd51e670fb2eec46b3cc96a656fdf45abbfce4d5bb74d137f541e8fda6b3"}
Oct 30 20:20:24.359891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:24.359788    4519 generic.go:356] "Generic (PLEG): container finished" podID="8e28ee8c-8795-4313-8b01-92845e6c40e3" containerID="dc42e71a54e3483d8d8dab770089d8c0805b36d323ee4ab7d9082753d2424494" exitCode=0
Oct 30 20:20:24.360446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:24.359884    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb" event={"ID":"8e28ee8c-8795-4313-8b01-92845e6c40e3","Type":"ContainerDied","Data":"dc42e71a54e3483d8d8dab770089d8c0805b36d323ee4ab7d9082753d2424494"}
Oct 30 20:20:25.562623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.562597    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:25.696633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.696597    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-openshift-ca\") pod \"8e28ee8c-8795-4313-8b01-92845e6c40e3\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") "
Oct 30 20:20:25.696841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.696711    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e28ee8c-8795-4313-8b01-92845e6c40e3-kube-api-access-6x9cp\" (UniqueName: \"kubernetes.io/projected/8e28ee8c-8795-4313-8b01-92845e6c40e3-kube-api-access-6x9cp\") pod \"8e28ee8c-8795-4313-8b01-92845e6c40e3\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") "
Oct 30 20:20:25.696841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.696814    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-xks-ca\") pod \"8e28ee8c-8795-4313-8b01-92845e6c40e3\" (UID: \"8e28ee8c-8795-4313-8b01-92845e6c40e3\") "
Oct 30 20:20:25.697077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.697044    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-openshift-ca" pod "8e28ee8c-8795-4313-8b01-92845e6c40e3" (UID: "8e28ee8c-8795-4313-8b01-92845e6c40e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:25.698213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.698007    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:20:25.699516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.699488    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-xks-ca" pod "8e28ee8c-8795-4313-8b01-92845e6c40e3" (UID: "8e28ee8c-8795-4313-8b01-92845e6c40e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:25.700035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.700013    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e28ee8c-8795-4313-8b01-92845e6c40e3-kube-api-access-6x9cp" pod "8e28ee8c-8795-4313-8b01-92845e6c40e3" (UID: "8e28ee8c-8795-4313-8b01-92845e6c40e3"). InnerVolumeSpecName "kube-api-access-6x9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:25.799014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.798978    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e28ee8c-8795-4313-8b01-92845e6c40e3-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:20:25.799014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:25.799011    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9cp\" (UniqueName: \"kubernetes.io/projected/8e28ee8c-8795-4313-8b01-92845e6c40e3-kube-api-access-6x9cp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:20:26.382559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:26.382522    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb" event={"ID":"8e28ee8c-8795-4313-8b01-92845e6c40e3","Type":"ContainerDied","Data":"9500fd51e670fb2eec46b3cc96a656fdf45abbfce4d5bb74d137f541e8fda6b3"}
Oct 30 20:20:26.382773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:26.382574    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9500fd51e670fb2eec46b3cc96a656fdf45abbfce4d5bb74d137f541e8fda6b3"
Oct 30 20:20:26.382773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:26.382573    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"
Oct 30 20:20:32.224142 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:20:32.224099    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:20:44.508990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:44.508955    4519 scope.go:122] "RemoveContainer" containerID="0db15f5b0b09adfe3ef11605f008fc0daea63efe85bd09f47d1a382788909862"
Oct 30 20:20:44.522332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:44.522305    4519 scope.go:122] "RemoveContainer" containerID="c56914748c6e51d94cb8c56154ef90c0dc80ff5e3868b425995bb0ef40b0c63e"
Oct 30 20:20:44.533656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:44.533635    4519 scope.go:122] "RemoveContainer" containerID="0531bfa8522d26b66f861d28fc8fe6965f699530555603de92dd6fc9f9267b78"
Oct 30 20:20:44.545804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:44.545777    4519 scope.go:122] "RemoveContainer" containerID="c89c0f50a149a47847e0d153663d0fa208cdc09bc623285a014ff7497bfe7e79"
Oct 30 20:20:44.558846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:44.558821    4519 scope.go:122] "RemoveContainer" containerID="17feb76aefd77e129980cfae6620591190e3af9254866b0ec1510ba6aadf4f5e"
Oct 30 20:20:44.569403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:44.569377    4519 scope.go:122] "RemoveContainer" containerID="23e965ba54a93fd131dbdbf8d4504c2732c287cec4d2a8ada8ba8b577434a931"
Oct 30 20:20:46.198535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:20:46.186161    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"]
Oct 30 20:21:01.031431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.031388    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"]
Oct 30 20:21:01.044747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.044713    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37048f8-f4xxl"]
Oct 30 20:21:01.216247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.216191    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l" podUID="5d841ab1-9f09-4836-894d-8fa3a3cb3803" containerName="main" containerID="cri-o://a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062" gracePeriod=45
Oct 30 20:21:01.520222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.520194    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:21:01.615928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.615833    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb" path="/var/lib/kubelet/pods/0c715fa8-6f9b-4deb-a6ae-8769ae6ed7bb/volumes"
Oct 30 20:21:01.656230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656195    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tmp-dir\") pod \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") "
Oct 30 20:21:01.656437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656257    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-home\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-home\") pod \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") "
Oct 30 20:21:01.656437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656288    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-dshm\") pod \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") "
Oct 30 20:21:01.656437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656427    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-model-cache\") pod \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") "
Oct 30 20:21:01.656671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656477    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d841ab1-9f09-4836-894d-8fa3a3cb3803-kube-api-access-jx7vc\" (UniqueName: \"kubernetes.io/projected/5d841ab1-9f09-4836-894d-8fa3a3cb3803-kube-api-access-jx7vc\") pod \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") "
Oct 30 20:21:01.656671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656511    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tmp-dir" pod "5d841ab1-9f09-4836-894d-8fa3a3cb3803" (UID: "5d841ab1-9f09-4836-894d-8fa3a3cb3803"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:01.656671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656607    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-home" pod "5d841ab1-9f09-4836-894d-8fa3a3cb3803" (UID: "5d841ab1-9f09-4836-894d-8fa3a3cb3803"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:01.656671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.656611    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tls-certs\") pod \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\" (UID: \"5d841ab1-9f09-4836-894d-8fa3a3cb3803\") "
Oct 30 20:21:01.657117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.657089    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-model-cache" pod "5d841ab1-9f09-4836-894d-8fa3a3cb3803" (UID: "5d841ab1-9f09-4836-894d-8fa3a3cb3803"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:01.657292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.657268    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:01.657402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.657297    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:01.658940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.658908    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tls-certs" pod "5d841ab1-9f09-4836-894d-8fa3a3cb3803" (UID: "5d841ab1-9f09-4836-894d-8fa3a3cb3803"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:01.659290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.659270    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-dshm" pod "5d841ab1-9f09-4836-894d-8fa3a3cb3803" (UID: "5d841ab1-9f09-4836-894d-8fa3a3cb3803"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:01.659449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.659421    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d841ab1-9f09-4836-894d-8fa3a3cb3803-kube-api-access-jx7vc" pod "5d841ab1-9f09-4836-894d-8fa3a3cb3803" (UID: "5d841ab1-9f09-4836-894d-8fa3a3cb3803"). InnerVolumeSpecName "kube-api-access-jx7vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:01.743691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.743652    4519 generic.go:356] "Generic (PLEG): container finished" podID="5d841ab1-9f09-4836-894d-8fa3a3cb3803" containerID="a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062" exitCode=0
Oct 30 20:21:01.743851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.743701    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"
Oct 30 20:21:01.743851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.743737    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l" event={"ID":"5d841ab1-9f09-4836-894d-8fa3a3cb3803","Type":"ContainerDied","Data":"a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062"}
Oct 30 20:21:01.743851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.743786    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l" event={"ID":"5d841ab1-9f09-4836-894d-8fa3a3cb3803","Type":"ContainerDied","Data":"cb3ec2f4a6780096522e88cab1c886c19dd0e7dc6efaff0292c2ea42800634f7"}
Oct 30 20:21:01.743851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.743805    4519 scope.go:122] "RemoveContainer" containerID="a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062"
Oct 30 20:21:01.759898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.759856    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:01.759898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.759891    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d841ab1-9f09-4836-894d-8fa3a3cb3803-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:01.760116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.759906    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7vc\" (UniqueName: \"kubernetes.io/projected/5d841ab1-9f09-4836-894d-8fa3a3cb3803-kube-api-access-jx7vc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:01.760116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.759920    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d841ab1-9f09-4836-894d-8fa3a3cb3803-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:01.764472 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.764445    4519 scope.go:122] "RemoveContainer" containerID="a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062"
Oct 30 20:21:01.764807 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:21:01.764784    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062\": container with ID starting with a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062 not found: ID does not exist" containerID="a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062"
Oct 30 20:21:01.764912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.764816    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062"} err="failed to get container status \"a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062\": rpc error: code = NotFound desc = could not find container \"a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062\": container with ID starting with a77e81e470285bb405f920f6d5ec149d63cbd6b03494f5187777482807cac062 not found: ID does not exist"
Oct 30 20:21:01.800186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.800154    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"]
Oct 30 20:21:01.804318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:01.804290    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a165e5/auth-test-model-a165e5-kserve-58988f865-8x96l"]
Oct 30 20:21:02.223831 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:21:02.223786    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:21:03.614091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:03.614058    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d841ab1-9f09-4836-894d-8fa3a3cb3803" path="/var/lib/kubelet/pods/5d841ab1-9f09-4836-894d-8fa3a3cb3803/volumes"
Oct 30 20:21:06.415455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:06.415416    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"]
Oct 30 20:21:07.549505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.542673    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"]
Oct 30 20:21:07.549505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.543600    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5d841ab1-9f09-4836-894d-8fa3a3cb3803" containerName="main"
Oct 30 20:21:07.549505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.543625    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d841ab1-9f09-4836-894d-8fa3a3cb3803" containerName="main"
Oct 30 20:21:07.549505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.543680    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8e28ee8c-8795-4313-8b01-92845e6c40e3" containerName="revoke-keys"
Oct 30 20:21:07.549505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.543689    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e28ee8c-8795-4313-8b01-92845e6c40e3" containerName="revoke-keys"
Oct 30 20:21:07.549505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.543854    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d841ab1-9f09-4836-894d-8fa3a3cb3803" containerName="main"
Oct 30 20:21:07.549505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.543876    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e28ee8c-8795-4313-8b01-92845e6c40e3" containerName="revoke-keys"
Oct 30 20:21:07.550811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.550784    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558255    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-token\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558358    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-data\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558422    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-socket\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558459    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-certs\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558535    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-podinfo\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558568    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-envoy\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558608    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vkx\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-kube-api-access-g7vkx\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558670    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istiod-ca-cert\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.558947 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.558754    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-credential-socket\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.582747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.582714    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5fbda3-openshift-default-dockercfg-8fccx\""
Oct 30 20:21:07.588024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.587997    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"]
Oct 30 20:21:07.660179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660139    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istiod-ca-cert\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660213    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-credential-socket\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660278    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-token\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660340    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-data\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660395    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-socket\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660429    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-certs\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660505    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-podinfo\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660540    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-envoy\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660579    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vkx\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-kube-api-access-g7vkx\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660774    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-credential-socket\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.660889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.660802    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-socket\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.661101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.661077    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-certs\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.661164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.661098    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-data\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.661662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.661639    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istiod-ca-cert\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.663598 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.663570    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-envoy\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.663892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.663868    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-podinfo\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.679974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.679938    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-token\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.680336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.680313    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vkx\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-kube-api-access-g7vkx\") pod \"e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") " pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:07.883944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:07.883784    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:08.623769 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:21:08.623731    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3085fd_c4f8_431f_a214_6ad1bfd1288e.slice/crio-60e56dbcd77720b29f1f13d4d92f42149d36ed404b017b07a42ba5dc53eb9d7c WatchSource:0}: Error finding container 60e56dbcd77720b29f1f13d4d92f42149d36ed404b017b07a42ba5dc53eb9d7c: Status 404 returned error can't find the container with id 60e56dbcd77720b29f1f13d4d92f42149d36ed404b017b07a42ba5dc53eb9d7c
Oct 30 20:21:08.624213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.624189    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"]
Oct 30 20:21:08.627137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.627094    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:08.627239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.627184    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:08.627239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.627226    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:08.832045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.831995    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" event={"ID":"ee3085fd-c4f8-431f-a214-6ad1bfd1288e","Type":"ContainerStarted","Data":"5baf4cdf932b7509d8bf981fce56eb327804561cf15aef3a2a1fd55637dfc400"}
Oct 30 20:21:08.832213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.832074    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" event={"ID":"ee3085fd-c4f8-431f-a214-6ad1bfd1288e","Type":"ContainerStarted","Data":"60e56dbcd77720b29f1f13d4d92f42149d36ed404b017b07a42ba5dc53eb9d7c"}
Oct 30 20:21:08.884545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.884063    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:08.896665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:08.896601    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" podStartSLOduration=1.8965852669999999 podStartE2EDuration="1.896585267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:08.894053606 +0000 UTC m=+2793.394757334" watchObservedRunningTime="2026-10-30 20:21:08.896585267 +0000 UTC m=+2793.397288990"
Oct 30 20:21:09.891588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:09.891546    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:10.849538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:10.849498    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:10.850555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:10.850528    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:21:21.445799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.445748    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh" podUID="3cf1e755-9c05-48de-8b31-102f7afa9e23" containerName="main" containerID="cri-o://9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113" gracePeriod=45
Oct 30 20:21:21.766721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.766689    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:21:21.910300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910261    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-model-cache\") pod \"3cf1e755-9c05-48de-8b31-102f7afa9e23\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") "
Oct 30 20:21:21.910553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910330    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-tmp-dir\") pod \"3cf1e755-9c05-48de-8b31-102f7afa9e23\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") "
Oct 30 20:21:21.910553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910444    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-home\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-home\") pod \"3cf1e755-9c05-48de-8b31-102f7afa9e23\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") "
Oct 30 20:21:21.910714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910588    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-dshm\") pod \"3cf1e755-9c05-48de-8b31-102f7afa9e23\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") "
Oct 30 20:21:21.910714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910608    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-tmp-dir" pod "3cf1e755-9c05-48de-8b31-102f7afa9e23" (UID: "3cf1e755-9c05-48de-8b31-102f7afa9e23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:21.910714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910621    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-model-cache" pod "3cf1e755-9c05-48de-8b31-102f7afa9e23" (UID: "3cf1e755-9c05-48de-8b31-102f7afa9e23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:21.910714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910643    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-home" pod "3cf1e755-9c05-48de-8b31-102f7afa9e23" (UID: "3cf1e755-9c05-48de-8b31-102f7afa9e23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:21.910714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910653    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf1e755-9c05-48de-8b31-102f7afa9e23-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf1e755-9c05-48de-8b31-102f7afa9e23-tls-certs\") pod \"3cf1e755-9c05-48de-8b31-102f7afa9e23\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") "
Oct 30 20:21:21.910714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.910696    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf1e755-9c05-48de-8b31-102f7afa9e23-kube-api-access-twt5x\" (UniqueName: \"kubernetes.io/projected/3cf1e755-9c05-48de-8b31-102f7afa9e23-kube-api-access-twt5x\") pod \"3cf1e755-9c05-48de-8b31-102f7afa9e23\" (UID: \"3cf1e755-9c05-48de-8b31-102f7afa9e23\") "
Oct 30 20:21:21.911322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.911302    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:21.911409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.911326    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:21.911409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.911338    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:21.912893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.912863    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-dshm" pod "3cf1e755-9c05-48de-8b31-102f7afa9e23" (UID: "3cf1e755-9c05-48de-8b31-102f7afa9e23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:21.912997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.912919    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf1e755-9c05-48de-8b31-102f7afa9e23-tls-certs" pod "3cf1e755-9c05-48de-8b31-102f7afa9e23" (UID: "3cf1e755-9c05-48de-8b31-102f7afa9e23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:21.912997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.912981    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf1e755-9c05-48de-8b31-102f7afa9e23-kube-api-access-twt5x" pod "3cf1e755-9c05-48de-8b31-102f7afa9e23" (UID: "3cf1e755-9c05-48de-8b31-102f7afa9e23"). InnerVolumeSpecName "kube-api-access-twt5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:21.974259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.974220    4519 generic.go:356] "Generic (PLEG): container finished" podID="3cf1e755-9c05-48de-8b31-102f7afa9e23" containerID="9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113" exitCode=0
Oct 30 20:21:21.974449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.974296    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"
Oct 30 20:21:21.974449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.974314    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh" event={"ID":"3cf1e755-9c05-48de-8b31-102f7afa9e23","Type":"ContainerDied","Data":"9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113"}
Oct 30 20:21:21.974449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.974367    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh" event={"ID":"3cf1e755-9c05-48de-8b31-102f7afa9e23","Type":"ContainerDied","Data":"82f99142fed68c4d67ef196e2660c855c79d11631f81a38a955e186f7e8818ef"}
Oct 30 20:21:21.974449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.974390    4519 scope.go:122] "RemoveContainer" containerID="9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113"
Oct 30 20:21:21.987997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.987948    4519 scope.go:122] "RemoveContainer" containerID="9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113"
Oct 30 20:21:21.988319 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:21:21.988298    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113\": container with ID starting with 9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113 not found: ID does not exist" containerID="9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113"
Oct 30 20:21:21.988394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:21.988328    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113"} err="failed to get container status \"9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113\": rpc error: code = NotFound desc = could not find container \"9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113\": container with ID starting with 9aae12f77133ac53afc8f93bf53b8a5a94f10477f04f88d5e265b43d7cdf9113 not found: ID does not exist"
Oct 30 20:21:22.013084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:22.013056    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf1e755-9c05-48de-8b31-102f7afa9e23-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:22.013084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:22.013082    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf1e755-9c05-48de-8b31-102f7afa9e23-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:22.013239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:22.013093    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twt5x\" (UniqueName: \"kubernetes.io/projected/3cf1e755-9c05-48de-8b31-102f7afa9e23-kube-api-access-twt5x\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:21:22.029381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:22.029344    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"]
Oct 30 20:21:22.037729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:22.037706    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-87ff9f/auth-test-model-87ff9f-kserve-5c4d6877cc-x57xh"]
Oct 30 20:21:23.613981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:23.613941    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf1e755-9c05-48de-8b31-102f7afa9e23" path="/var/lib/kubelet/pods/3cf1e755-9c05-48de-8b31-102f7afa9e23/volumes"
Oct 30 20:21:30.031969 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:30.031929    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"]
Oct 30 20:21:30.044240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:30.044210    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cb8475-2djbx"]
Oct 30 20:21:31.613336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:31.613299    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1" path="/var/lib/kubelet/pods/fdc5b6a6-e216-41dd-8fd1-3fb35a12aae1/volumes"
Oct 30 20:21:32.223487 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:21:32.223437    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:21:33.032683 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:33.032639    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"]
Oct 30 20:21:33.044234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:33.044203    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59e7611d-4kjfr"]
Oct 30 20:21:33.614820 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:33.614770    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9" path="/var/lib/kubelet/pods/72a6cd71-e9f4-4cde-b7de-5e0bf4e867c9/volumes"
Oct 30 20:21:44.032377 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:44.032338    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"]
Oct 30 20:21:44.044132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:44.044100    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94268f94-99595"]
Oct 30 20:21:44.626028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:44.625989    4519 scope.go:122] "RemoveContainer" containerID="6a9d65c09d200d41131ba0f295f545d4db67b3f57cb39e3bd483bc0e7eb92270"
Oct 30 20:21:44.636582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:44.636559    4519 scope.go:122] "RemoveContainer" containerID="c43978cdf40727894416c6dc1a552bf20648209e92bad5e2fdfa6fc161cd4dc5"
Oct 30 20:21:45.612861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:45.612818    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f94e19-8134-4cf0-8815-462f1a84034a" path="/var/lib/kubelet/pods/42f94e19-8134-4cf0-8815-462f1a84034a/volumes"
Oct 30 20:21:46.932353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.932313    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"]
Oct 30 20:21:46.933222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.933199    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3cf1e755-9c05-48de-8b31-102f7afa9e23" containerName="main"
Oct 30 20:21:46.933290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.933225    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf1e755-9c05-48de-8b31-102f7afa9e23" containerName="main"
Oct 30 20:21:46.933453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.933438    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf1e755-9c05-48de-8b31-102f7afa9e23" containerName="main"
Oct 30 20:21:46.938770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.938742    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:46.962308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.962280    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6175d5\""
Oct 30 20:21:46.963319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.963293    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6175d5-dockercfg-8rx8f\""
Oct 30 20:21:46.981323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:46.981293    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"]
Oct 30 20:21:47.038081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.038046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6hj\" (UniqueName: \"kubernetes.io/projected/98453c2c-4cb9-482f-9638-55553dfc6e8e-kube-api-access-gd6hj\") pod \"payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:47.038290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.038219    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98453c2c-4cb9-482f-9638-55553dfc6e8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:47.140063 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.140024    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6hj\" (UniqueName: \"kubernetes.io/projected/98453c2c-4cb9-482f-9638-55553dfc6e8e-kube-api-access-gd6hj\") pod \"payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:47.140271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.140112    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98453c2c-4cb9-482f-9638-55553dfc6e8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:47.141061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.141035    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98453c2c-4cb9-482f-9638-55553dfc6e8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:47.160007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.159973    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6hj\" (UniqueName: \"kubernetes.io/projected/98453c2c-4cb9-482f-9638-55553dfc6e8e-kube-api-access-gd6hj\") pod \"payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:47.294289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.294209    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:47.602049 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:21:47.602007    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98453c2c_4cb9_482f_9638_55553dfc6e8e.slice/crio-f62438518052274216e3f7c1e31aeb6213ef86e544d7b71609af2b46465fa871 WatchSource:0}: Error finding container f62438518052274216e3f7c1e31aeb6213ef86e544d7b71609af2b46465fa871: Status 404 returned error can't find the container with id f62438518052274216e3f7c1e31aeb6213ef86e544d7b71609af2b46465fa871
Oct 30 20:21:47.602531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:47.602508    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"]
Oct 30 20:21:48.250008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:48.249965    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt" event={"ID":"98453c2c-4cb9-482f-9638-55553dfc6e8e","Type":"ContainerStarted","Data":"cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6"}
Oct 30 20:21:48.250528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:48.250025    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt" event={"ID":"98453c2c-4cb9-482f-9638-55553dfc6e8e","Type":"ContainerStarted","Data":"f62438518052274216e3f7c1e31aeb6213ef86e544d7b71609af2b46465fa871"}
Oct 30 20:21:48.250528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:48.250053    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:21:48.292377 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:48.292314    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt" podStartSLOduration=2.292300783 podStartE2EDuration="2.292300783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:48.289265957 +0000 UTC m=+2832.789969680" watchObservedRunningTime="2026-10-30 20:21:48.292300783 +0000 UTC m=+2832.793004506"
Oct 30 20:21:59.262203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:21:59.262167    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:22:00.798359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.798313    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"]
Oct 30 20:22:00.818489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.818436    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.844501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.838954    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b7eec8-openshift-default-dockercfg-46fn2\""
Oct 30 20:22:00.920504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.908296    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-credential-socket\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.940134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.940090    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-certs\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.957667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.957634    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26412140-b8b8-428b-802f-4e1a66f8fd79-istiod-ca-cert\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.957922 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.957900    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-envoy\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.958163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.958141    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-podinfo\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.958332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.958316    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-token\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.958497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.958460    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-data\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.958669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.958647    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfc2f\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-kube-api-access-zfc2f\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:00.958976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:00.958951    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-socket\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.038525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.026254    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"]
Oct 30 20:22:01.077600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.064431    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-credential-socket\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.080641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.080608    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-certs\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.080847 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.080833    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26412140-b8b8-428b-802f-4e1a66f8fd79-istiod-ca-cert\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.080985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.080969    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-envoy\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.081172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.081158    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-podinfo\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.081294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.081282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-token\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.081430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.081417    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-data\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.081549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.081534    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfc2f\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-kube-api-access-zfc2f\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.081808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.081794    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-socket\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.082337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.082314    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-socket\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.082501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.067507    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-credential-socket\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.082911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.082894    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-certs\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.083635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.083593    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26412140-b8b8-428b-802f-4e1a66f8fd79-istiod-ca-cert\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.090529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.085556    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-data\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.099714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.097995    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-podinfo\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.126330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.102981    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-envoy\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.127114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.126876    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-token\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.130003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.129976    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfc2f\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-kube-api-access-zfc2f\") pod \"e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") " pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.397671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.397563    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:01.771720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.771684    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"]
Oct 30 20:22:01.773735 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:22:01.773696    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26412140_b8b8_428b_802f_4e1a66f8fd79.slice/crio-ab23335e3b6d7738d1b9e82ca94d2e3695918dd546d0d976fcd69a75432eda86 WatchSource:0}: Error finding container ab23335e3b6d7738d1b9e82ca94d2e3695918dd546d0d976fcd69a75432eda86: Status 404 returned error can't find the container with id ab23335e3b6d7738d1b9e82ca94d2e3695918dd546d0d976fcd69a75432eda86
Oct 30 20:22:01.777411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.777366    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:22:01.777581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.777484    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:22:01.777581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:01.777533    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:22:02.227392 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:22:02.227358    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:22:02.378488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:02.377915    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" event={"ID":"26412140-b8b8-428b-802f-4e1a66f8fd79","Type":"ContainerStarted","Data":"3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f"}
Oct 30 20:22:02.378488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:02.377979    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" event={"ID":"26412140-b8b8-428b-802f-4e1a66f8fd79","Type":"ContainerStarted","Data":"ab23335e3b6d7738d1b9e82ca94d2e3695918dd546d0d976fcd69a75432eda86"}
Oct 30 20:22:02.398713 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:02.398676    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:02.405137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:02.405109    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:02.445842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:02.445781    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podStartSLOduration=2.445759647 podStartE2EDuration="2.445759647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:02.43386433 +0000 UTC m=+2846.934568055" watchObservedRunningTime="2026-10-30 20:22:02.445759647 +0000 UTC m=+2846.946463373"
Oct 30 20:22:03.388336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:03.388295    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:03.389305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:03.389281    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:22:22.699632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.699592    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"]
Oct 30 20:22:22.703984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.703955    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.732520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.732484    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-188d45\"/\"openshift-service-ca.crt\""
Oct 30 20:22:22.732759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.732532    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-188d45\"/\"default-dockercfg-6c7xv\""
Oct 30 20:22:22.732759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.732532    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-188d45\"/\"kube-root-ca.crt\""
Oct 30 20:22:22.732991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.732644    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-188d45\"/\"ipp-route-188d45-kserve-self-signed-certs\""
Oct 30 20:22:22.738975 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.738943    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"]
Oct 30 20:22:22.761560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.761531    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-dshm\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.761940 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.761913    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-model-cache\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.762161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.762137    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-home\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.762375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.762356    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tmp-dir\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.762556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.762531    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tls-certs\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.762787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.762761    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfs5s\" (UniqueName: \"kubernetes.io/projected/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-kube-api-access-sfs5s\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.864295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.864254    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-home\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.864295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.864301    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tmp-dir\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.864599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.864332    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tls-certs\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.864599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.864433    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfs5s\" (UniqueName: \"kubernetes.io/projected/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-kube-api-access-sfs5s\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.864599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.864553    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-dshm\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.864764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.864716    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-home\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.864826 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.864786    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-model-cache\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.865185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.865158    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-model-cache\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.865800 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.865719    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tmp-dir\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.867882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.867839    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-dshm\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.868933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.868912    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tls-certs\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:22.888511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:22.885128    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfs5s\" (UniqueName: \"kubernetes.io/projected/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-kube-api-access-sfs5s\") pod \"ipp-route-188d45-kserve-c85cbfd85-wqd8l\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") " pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:23.024544 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:23.021743    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:23.444962 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:22:23.444927    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e8bce9e_0347_49ee_b0f0_ac1f2b0cfca2.slice/crio-44d97fb2e1cd17d7af9b36ec8f8d08582b1a600cba54b91c21cfd0770d873786 WatchSource:0}: Error finding container 44d97fb2e1cd17d7af9b36ec8f8d08582b1a600cba54b91c21cfd0770d873786: Status 404 returned error can't find the container with id 44d97fb2e1cd17d7af9b36ec8f8d08582b1a600cba54b91c21cfd0770d873786
Oct 30 20:22:23.445863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:23.445828    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"]
Oct 30 20:22:23.596932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:23.596885    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l" event={"ID":"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2","Type":"ContainerStarted","Data":"06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9"}
Oct 30 20:22:23.597121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:23.596941    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l" event={"ID":"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2","Type":"ContainerStarted","Data":"44d97fb2e1cd17d7af9b36ec8f8d08582b1a600cba54b91c21cfd0770d873786"}
Oct 30 20:22:23.639923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:23.639861    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l" podStartSLOduration=1.639841574 podStartE2EDuration="1.639841574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:23.638740286 +0000 UTC m=+2868.139444010" watchObservedRunningTime="2026-10-30 20:22:23.639841574 +0000 UTC m=+2868.140545300"
Oct 30 20:22:24.022871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:24.022835    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:32.222902 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:22:32.222860    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:22:33.024491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:33.023888    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:33.110491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:33.110110    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:33.718970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:33.718938    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:22:44.686990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.686952    4519 scope.go:122] "RemoveContainer" containerID="b2478f27ab21f6eaa6e08ec6d3452bdeff670c2b0e86d335b7c7dd7afe67a7d5"
Oct 30 20:22:44.699567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.699541    4519 scope.go:122] "RemoveContainer" containerID="c521ae4d4ccdf24d0a310c188a4ae05ba69b08257d58f74867e96c204a42378d"
Oct 30 20:22:44.709688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.709662    4519 scope.go:122] "RemoveContainer" containerID="17e797cd2f1eec221fa4b7cabb1669cf4686cd9b485791fb490e58b24d1fe357"
Oct 30 20:22:44.721398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.721375    4519 scope.go:122] "RemoveContainer" containerID="6d76cb83ddda7443df947aba12885c257e5a6ceccb55ba6101c1793c147ce4dd"
Oct 30 20:22:44.950833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.950738    4519 generic.go:356] "Generic (PLEG): container finished" podID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerID="5baf4cdf932b7509d8bf981fce56eb327804561cf15aef3a2a1fd55637dfc400" exitCode=0
Oct 30 20:22:44.950833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.950791    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" event={"ID":"ee3085fd-c4f8-431f-a214-6ad1bfd1288e","Type":"ContainerDied","Data":"5baf4cdf932b7509d8bf981fce56eb327804561cf15aef3a2a1fd55637dfc400"}
Oct 30 20:22:44.951227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.951211    4519 scope.go:122] "RemoveContainer" containerID="5baf4cdf932b7509d8bf981fce56eb327804561cf15aef3a2a1fd55637dfc400"
Oct 30 20:22:44.953179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.953150    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:22:44.953267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.953223    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:22:44.953267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:44.953258    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:22:45.979534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:45.979495    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" event={"ID":"ee3085fd-c4f8-431f-a214-6ad1bfd1288e","Type":"ContainerStarted","Data":"bfdd5282829684254f1c5043db656528f671cb3f737ac15a703ae406d6546dcf"}
Oct 30 20:22:46.888251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:46.887652    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 20:22:46.888251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:46.887735    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 20:22:46.888251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:46.888028    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:22:47.035266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:47.035227    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"]
Oct 30 20:22:47.071490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:47.061807    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a4035d-2ltdl"]
Oct 30 20:22:47.613677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:47.613637    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ad26ac-3997-441f-9f79-5c4253e128a7" path="/var/lib/kubelet/pods/21ad26ac-3997-441f-9f79-5c4253e128a7/volumes"
Oct 30 20:22:47.885102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:47.885016    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 20:22:47.885263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:47.885086    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 20:22:48.884926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:48.884856    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 20:22:48.884926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:48.884923    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 20:22:49.885145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:49.885101    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 20:22:49.885682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:49.885165    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 20:22:50.884733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:50.884688    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 20:22:50.885058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:50.885023    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 20:22:51.890615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:51.890575    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:22:51.891132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:51.891018    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:22:51.891655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:51.891624    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:22:59.033702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:59.033666    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"]
Oct 30 20:22:59.045717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:59.045684    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dd013d-gcdn9"]
Oct 30 20:22:59.613552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:22:59.613513    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5387407-2f32-4cdc-83c4-fe88663933db" path="/var/lib/kubelet/pods/c5387407-2f32-4cdc-83c4-fe88663933db/volumes"
Oct 30 20:23:02.222725 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:02.222683    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:23:07.704375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.704332    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"]
Oct 30 20:23:07.708752 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.708728    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.728389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.728355    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"]
Oct 30 20:23:07.734709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.734676    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:07.734909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.734742    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:07.761664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.761630    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.761859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.761811    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.761967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.761947    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbxg\" (UniqueName: \"kubernetes.io/projected/bfefa741-d9ab-48fb-aad8-e32ef65182c2-kube-api-access-7hbxg\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.863539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.863497    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.863967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.863945    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbxg\" (UniqueName: \"kubernetes.io/projected/bfefa741-d9ab-48fb-aad8-e32ef65182c2-kube-api-access-7hbxg\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.864170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.864155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.864349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.864320    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.867407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.867377    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:07.883563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:07.883532    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbxg\" (UniqueName: \"kubernetes.io/projected/bfefa741-d9ab-48fb-aad8-e32ef65182c2-kube-api-access-7hbxg\") pod \"maas-api-revoke-sub-1f956601-jhbrd\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:08.024706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.024613    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:08.232866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.232823    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7059-999173"]
Oct 30 20:23:08.237280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.237254    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:08.251818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.251793    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-brtz2\""
Oct 30 20:23:08.252295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.252270    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-999173"]
Oct 30 20:23:08.271635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.271600    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2nx\" (UniqueName: \"kubernetes.io/projected/258fd0ef-38fa-4ac8-bebe-e53784405961-kube-api-access-dr2nx\") pod \"test-curl-7059-999173\" (UID: \"258fd0ef-38fa-4ac8-bebe-e53784405961\") " pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:08.339547 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:08.339459    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfefa741_d9ab_48fb_aad8_e32ef65182c2.slice/crio-de173ce1e142f89aa5e1eeffc460584d83f53df224f4fbda2450279f8eae36e8 WatchSource:0}: Error finding container de173ce1e142f89aa5e1eeffc460584d83f53df224f4fbda2450279f8eae36e8: Status 404 returned error can't find the container with id de173ce1e142f89aa5e1eeffc460584d83f53df224f4fbda2450279f8eae36e8
Oct 30 20:23:08.342921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.342895    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"]
Oct 30 20:23:08.373436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.373404    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2nx\" (UniqueName: \"kubernetes.io/projected/258fd0ef-38fa-4ac8-bebe-e53784405961-kube-api-access-dr2nx\") pod \"test-curl-7059-999173\" (UID: \"258fd0ef-38fa-4ac8-bebe-e53784405961\") " pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:08.393200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.393157    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2nx\" (UniqueName: \"kubernetes.io/projected/258fd0ef-38fa-4ac8-bebe-e53784405961-kube-api-access-dr2nx\") pod \"test-curl-7059-999173\" (UID: \"258fd0ef-38fa-4ac8-bebe-e53784405961\") " pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:08.553684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.553647    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:08.860913 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:08.860839    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod258fd0ef_38fa_4ac8_bebe_e53784405961.slice/crio-5c91d7d6901ecaa8360d851d69633c8886e51a2501c38d2b0692002bb620a010 WatchSource:0}: Error finding container 5c91d7d6901ecaa8360d851d69633c8886e51a2501c38d2b0692002bb620a010: Status 404 returned error can't find the container with id 5c91d7d6901ecaa8360d851d69633c8886e51a2501c38d2b0692002bb620a010
Oct 30 20:23:08.862037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:08.862005    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-999173"]
Oct 30 20:23:09.308777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:09.308735    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-999173" event={"ID":"258fd0ef-38fa-4ac8-bebe-e53784405961","Type":"ContainerStarted","Data":"f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f"}
Oct 30 20:23:09.308999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:09.308789    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-999173" event={"ID":"258fd0ef-38fa-4ac8-bebe-e53784405961","Type":"ContainerStarted","Data":"5c91d7d6901ecaa8360d851d69633c8886e51a2501c38d2b0692002bb620a010"}
Oct 30 20:23:09.310442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:09.310416    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd" event={"ID":"bfefa741-d9ab-48fb-aad8-e32ef65182c2","Type":"ContainerStarted","Data":"707a36453b8e476288bc63b4eab09416bb859bd376d039dc6790bb74fbe256e7"}
Oct 30 20:23:09.310599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:09.310498    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd" event={"ID":"bfefa741-d9ab-48fb-aad8-e32ef65182c2","Type":"ContainerStarted","Data":"de173ce1e142f89aa5e1eeffc460584d83f53df224f4fbda2450279f8eae36e8"}
Oct 30 20:23:09.351422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:09.351364    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7059-999173" podStartSLOduration=1.351350689 podStartE2EDuration="1.351350689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:09.346022457 +0000 UTC m=+2913.846726182" watchObservedRunningTime="2026-10-30 20:23:09.351350689 +0000 UTC m=+2913.852054413"
Oct 30 20:23:10.257897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.257832    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd" podStartSLOduration=3.257816318 podStartE2EDuration="3.257816318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:09.391699629 +0000 UTC m=+2913.892403354" watchObservedRunningTime="2026-10-30 20:23:10.257816318 +0000 UTC m=+2914.758520082"
Oct 30 20:23:10.259982 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.259946    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7059-999173"]
Oct 30 20:23:10.264477 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.264433    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7059-999173"]
Oct 30 20:23:10.320698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.320662    4519 generic.go:356] "Generic (PLEG): container finished" podID="bfefa741-d9ab-48fb-aad8-e32ef65182c2" containerID="707a36453b8e476288bc63b4eab09416bb859bd376d039dc6790bb74fbe256e7" exitCode=0
Oct 30 20:23:10.320894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.320746    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd" event={"ID":"bfefa741-d9ab-48fb-aad8-e32ef65182c2","Type":"ContainerDied","Data":"707a36453b8e476288bc63b4eab09416bb859bd376d039dc6790bb74fbe256e7"}
Oct 30 20:23:10.415305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.415264    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7059-65f1b6"]
Oct 30 20:23:10.416003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.415980    4519 cpu_manager.go:414] "containerMap: removing container" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" containerName="test-curl-7059-999173"
Oct 30 20:23:10.416003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.416002    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" containerName="test-curl-7059-999173"
Oct 30 20:23:10.416202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.416182    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" containerName="test-curl-7059-999173"
Oct 30 20:23:10.419953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.419931    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:10.435398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.435375    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-65f1b6"]
Oct 30 20:23:10.536401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.536307    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjfw6\" (UniqueName: \"kubernetes.io/projected/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d-kube-api-access-mjfw6\") pod \"test-curl-7059-65f1b6\" (UID: \"bad99fd9-5bde-4ec8-8d68-74e336bdfe3d\") " pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:10.638316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.638276    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjfw6\" (UniqueName: \"kubernetes.io/projected/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d-kube-api-access-mjfw6\") pod \"test-curl-7059-65f1b6\" (UID: \"bad99fd9-5bde-4ec8-8d68-74e336bdfe3d\") " pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:10.657633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.657595    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjfw6\" (UniqueName: \"kubernetes.io/projected/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d-kube-api-access-mjfw6\") pod \"test-curl-7059-65f1b6\" (UID: \"bad99fd9-5bde-4ec8-8d68-74e336bdfe3d\") " pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:10.734367 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:10.734325    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:11.037345 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:11.037297    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbad99fd9_5bde_4ec8_8d68_74e336bdfe3d.slice/crio-ecb84b1ad156971b959b2560bea4587f4023a668bb202d690ea8e85640ab3da7 WatchSource:0}: Error finding container ecb84b1ad156971b959b2560bea4587f4023a668bb202d690ea8e85640ab3da7: Status 404 returned error can't find the container with id ecb84b1ad156971b959b2560bea4587f4023a668bb202d690ea8e85640ab3da7
Oct 30 20:23:11.038937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.038908    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7059-65f1b6"]
Oct 30 20:23:11.334745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.331001    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-65f1b6" event={"ID":"bad99fd9-5bde-4ec8-8d68-74e336bdfe3d","Type":"ContainerStarted","Data":"4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee"}
Oct 30 20:23:11.334745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.331031    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7059-999173" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" containerName="test-curl-7059-999173" containerID="cri-o://f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f" gracePeriod=2
Oct 30 20:23:11.334745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.331062    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7059-65f1b6" event={"ID":"bad99fd9-5bde-4ec8-8d68-74e336bdfe3d","Type":"ContainerStarted","Data":"ecb84b1ad156971b959b2560bea4587f4023a668bb202d690ea8e85640ab3da7"}
Oct 30 20:23:11.385422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.385366    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7059-65f1b6" podStartSLOduration=1.385350831 podStartE2EDuration="1.385350831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:11.381516949 +0000 UTC m=+2915.882220732" watchObservedRunningTime="2026-10-30 20:23:11.385350831 +0000 UTC m=+2915.886054554"
Oct 30 20:23:11.395582 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:11.395539    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-999173\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:11.411502 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:11.409709    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-999173\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:11.536060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.536030    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:11.549760 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:11.549717    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-999173\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:11.654418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.654320    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-xks-ca\") pod \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") "
Oct 30 20:23:11.654612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.654511    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfefa741-d9ab-48fb-aad8-e32ef65182c2-kube-api-access-7hbxg\" (UniqueName: \"kubernetes.io/projected/bfefa741-d9ab-48fb-aad8-e32ef65182c2-kube-api-access-7hbxg\") pod \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") "
Oct 30 20:23:11.654612 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.654569    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-openshift-ca\") pod \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\" (UID: \"bfefa741-d9ab-48fb-aad8-e32ef65182c2\") "
Oct 30 20:23:11.655198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.655164    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-openshift-ca" pod "bfefa741-d9ab-48fb-aad8-e32ef65182c2" (UID: "bfefa741-d9ab-48fb-aad8-e32ef65182c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:11.655380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.655357    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:11.656737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.656703    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-xks-ca" pod "bfefa741-d9ab-48fb-aad8-e32ef65182c2" (UID: "bfefa741-d9ab-48fb-aad8-e32ef65182c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:11.656858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.656824    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfefa741-d9ab-48fb-aad8-e32ef65182c2-kube-api-access-7hbxg" pod "bfefa741-d9ab-48fb-aad8-e32ef65182c2" (UID: "bfefa741-d9ab-48fb-aad8-e32ef65182c2"). InnerVolumeSpecName "kube-api-access-7hbxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:11.757125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.757089    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbxg\" (UniqueName: \"kubernetes.io/projected/bfefa741-d9ab-48fb-aad8-e32ef65182c2-kube-api-access-7hbxg\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:11.757125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:11.757122    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfefa741-d9ab-48fb-aad8-e32ef65182c2-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:12.234959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.234905    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7059-65f1b6"]
Oct 30 20:23:12.255188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.255143    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7059-65f1b6"]
Oct 30 20:23:12.339889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.339852    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"
Oct 30 20:23:12.339889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.339851    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd" event={"ID":"bfefa741-d9ab-48fb-aad8-e32ef65182c2","Type":"ContainerDied","Data":"de173ce1e142f89aa5e1eeffc460584d83f53df224f4fbda2450279f8eae36e8"}
Oct 30 20:23:12.340431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.339951    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de173ce1e142f89aa5e1eeffc460584d83f53df224f4fbda2450279f8eae36e8"
Oct 30 20:23:12.801722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.801682    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"]
Oct 30 20:23:12.802330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.802310    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bfefa741-d9ab-48fb-aad8-e32ef65182c2" containerName="revoke-keys"
Oct 30 20:23:12.802454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.802332    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfefa741-d9ab-48fb-aad8-e32ef65182c2" containerName="revoke-keys"
Oct 30 20:23:12.802454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.802388    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" containerName="test-curl-7059-65f1b6"
Oct 30 20:23:12.802454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.802399    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" containerName="test-curl-7059-65f1b6"
Oct 30 20:23:12.802643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.802585    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" containerName="test-curl-7059-65f1b6"
Oct 30 20:23:12.802643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.802612    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfefa741-d9ab-48fb-aad8-e32ef65182c2" containerName="revoke-keys"
Oct 30 20:23:12.806344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.806320    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:12.821178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.821150    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:12.826590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.826563    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"]
Oct 30 20:23:12.832932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.832910    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:12.973385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.973336    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:12.973637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.973609    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:12.973784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:12.973756    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlc6p\" (UniqueName: \"kubernetes.io/projected/1a502c5e-7d1a-4489-9f51-52d525a03b94-kube-api-access-vlc6p\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.075149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.075054    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.075328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.075225    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.075328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.075312    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlc6p\" (UniqueName: \"kubernetes.io/projected/1a502c5e-7d1a-4489-9f51-52d525a03b94-kube-api-access-vlc6p\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.076000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.075971    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.077658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.077636    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.095750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.095719    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlc6p\" (UniqueName: \"kubernetes.io/projected/1a502c5e-7d1a-4489-9f51-52d525a03b94-kube-api-access-vlc6p\") pod \"maas-api-revoke-sub-4396420e-sz5ml\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.119923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.119901    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:13.349110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.349064    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7059-65f1b6" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" containerName="test-curl-7059-65f1b6" containerID="cri-o://4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee" gracePeriod=2
Oct 30 20:23:13.362473 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:13.362425    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-65f1b6\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:13.440570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.440531    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"]
Oct 30 20:23:13.442557 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:13.442522    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a502c5e_7d1a_4489_9f51_52d525a03b94.slice/crio-73739a529b5cc51b327f35467dae3f337bfc22421acc6acdaed696b53c454217 WatchSource:0}: Error finding container 73739a529b5cc51b327f35467dae3f337bfc22421acc6acdaed696b53c454217: Status 404 returned error can't find the container with id 73739a529b5cc51b327f35467dae3f337bfc22421acc6acdaed696b53c454217
Oct 30 20:23:13.508542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.508516    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:13.521768 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:13.521735    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-65f1b6\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:13.606053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.605963    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258fd0ef-38fa-4ac8-bebe-e53784405961-kube-api-access-dr2nx\" (UniqueName: \"kubernetes.io/projected/258fd0ef-38fa-4ac8-bebe-e53784405961-kube-api-access-dr2nx\") pod \"258fd0ef-38fa-4ac8-bebe-e53784405961\" (UID: \"258fd0ef-38fa-4ac8-bebe-e53784405961\") "
Oct 30 20:23:13.608202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.608176    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258fd0ef-38fa-4ac8-bebe-e53784405961-kube-api-access-dr2nx" pod "258fd0ef-38fa-4ac8-bebe-e53784405961" (UID: "258fd0ef-38fa-4ac8-bebe-e53784405961"). InnerVolumeSpecName "kube-api-access-dr2nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:13.613595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.613565    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258fd0ef-38fa-4ac8-bebe-e53784405961" path="/var/lib/kubelet/pods/258fd0ef-38fa-4ac8-bebe-e53784405961/volumes"
Oct 30 20:23:13.708204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:13.708168    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr2nx\" (UniqueName: \"kubernetes.io/projected/258fd0ef-38fa-4ac8-bebe-e53784405961-kube-api-access-dr2nx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:14.361770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:14.361735    4519 generic.go:356] "Generic (PLEG): container finished" podID="258fd0ef-38fa-4ac8-bebe-e53784405961" containerID="f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f" exitCode=137
Oct 30 20:23:14.362187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:14.361791    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-999173"
Oct 30 20:23:14.362187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:14.361845    4519 scope.go:122] "RemoveContainer" containerID="f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f"
Oct 30 20:23:14.364868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:14.364842    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml" event={"ID":"1a502c5e-7d1a-4489-9f51-52d525a03b94","Type":"ContainerStarted","Data":"8ee54d1f0f054104c2c312ec4a5aceaee49b1c733ff789aa88670d6f9e6066ce"}
Oct 30 20:23:14.365017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:14.364895    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml" event={"ID":"1a502c5e-7d1a-4489-9f51-52d525a03b94","Type":"ContainerStarted","Data":"73739a529b5cc51b327f35467dae3f337bfc22421acc6acdaed696b53c454217"}
Oct 30 20:23:14.431267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:14.431238    4519 scope.go:122] "RemoveContainer" containerID="f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f"
Oct 30 20:23:14.431711 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:14.431678    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f\": container with ID starting with f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f not found: ID does not exist" containerID="f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f"
Oct 30 20:23:14.431857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:14.431721    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f"} err="failed to get container status \"f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f\": rpc error: code = NotFound desc = could not find container \"f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f\": container with ID starting with f7924ff2c7e41ab3ab5cd55a7d9e700ac8a4506aa07bd1afc1394a085a52bf6f not found: ID does not exist"
Oct 30 20:23:15.376860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.376826    4519 generic.go:356] "Generic (PLEG): container finished" podID="1a502c5e-7d1a-4489-9f51-52d525a03b94" containerID="8ee54d1f0f054104c2c312ec4a5aceaee49b1c733ff789aa88670d6f9e6066ce" exitCode=0
Oct 30 20:23:15.377266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.376882    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml" event={"ID":"1a502c5e-7d1a-4489-9f51-52d525a03b94","Type":"ContainerDied","Data":"8ee54d1f0f054104c2c312ec4a5aceaee49b1c733ff789aa88670d6f9e6066ce"}
Oct 30 20:23:15.578326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.578295    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:15.603751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.603688    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:15.637873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.637826    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a502c5e-7d1a-4489-9f51-52d525a03b94-kube-api-access-vlc6p\" (UniqueName: \"kubernetes.io/projected/1a502c5e-7d1a-4489-9f51-52d525a03b94-kube-api-access-vlc6p\") pod \"1a502c5e-7d1a-4489-9f51-52d525a03b94\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") "
Oct 30 20:23:15.638074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.638033    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d-kube-api-access-mjfw6\" (UniqueName: \"kubernetes.io/projected/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d-kube-api-access-mjfw6\") pod \"bad99fd9-5bde-4ec8-8d68-74e336bdfe3d\" (UID: \"bad99fd9-5bde-4ec8-8d68-74e336bdfe3d\") "
Oct 30 20:23:15.638211 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.638194    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-openshift-ca\") pod \"1a502c5e-7d1a-4489-9f51-52d525a03b94\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") "
Oct 30 20:23:15.638363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.638344    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-xks-ca\") pod \"1a502c5e-7d1a-4489-9f51-52d525a03b94\" (UID: \"1a502c5e-7d1a-4489-9f51-52d525a03b94\") "
Oct 30 20:23:15.638660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.638615    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-openshift-ca" pod "1a502c5e-7d1a-4489-9f51-52d525a03b94" (UID: "1a502c5e-7d1a-4489-9f51-52d525a03b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:15.639273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.639108    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:15.640762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.640732    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a502c5e-7d1a-4489-9f51-52d525a03b94-kube-api-access-vlc6p" pod "1a502c5e-7d1a-4489-9f51-52d525a03b94" (UID: "1a502c5e-7d1a-4489-9f51-52d525a03b94"). InnerVolumeSpecName "kube-api-access-vlc6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:15.641053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.641031    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-xks-ca" pod "1a502c5e-7d1a-4489-9f51-52d525a03b94" (UID: "1a502c5e-7d1a-4489-9f51-52d525a03b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:15.641164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.641030    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d-kube-api-access-mjfw6" pod "bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" (UID: "bad99fd9-5bde-4ec8-8d68-74e336bdfe3d"). InnerVolumeSpecName "kube-api-access-mjfw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:15.740426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.740387    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjfw6\" (UniqueName: \"kubernetes.io/projected/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d-kube-api-access-mjfw6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:15.740426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.740419    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a502c5e-7d1a-4489-9f51-52d525a03b94-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:15.740426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:15.740431    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlc6p\" (UniqueName: \"kubernetes.io/projected/1a502c5e-7d1a-4489-9f51-52d525a03b94-kube-api-access-vlc6p\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:16.387113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.387080    4519 generic.go:356] "Generic (PLEG): container finished" podID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" containerID="4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee" exitCode=137
Oct 30 20:23:16.387657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.387143    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:16.387657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.387179    4519 scope.go:122] "RemoveContainer" containerID="4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee"
Oct 30 20:23:16.389028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.389008    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"
Oct 30 20:23:16.389170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.389034    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml" event={"ID":"1a502c5e-7d1a-4489-9f51-52d525a03b94","Type":"ContainerDied","Data":"73739a529b5cc51b327f35467dae3f337bfc22421acc6acdaed696b53c454217"}
Oct 30 20:23:16.389170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.389079    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73739a529b5cc51b327f35467dae3f337bfc22421acc6acdaed696b53c454217"
Oct 30 20:23:16.405682 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:16.400953    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-65f1b6\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:16.412258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.412218    4519 scope.go:122] "RemoveContainer" containerID="4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee"
Oct 30 20:23:16.412666 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:16.412637    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee\": container with ID starting with 4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee not found: ID does not exist" containerID="4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee"
Oct 30 20:23:16.412775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:16.412679    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee"} err="failed to get container status \"4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee\": rpc error: code = NotFound desc = could not find container \"4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee\": container with ID starting with 4b25513ff52d9b3fe7339f665d122f7154d3dd31a7408bd49110ff83a113a1ee not found: ID does not exist"
Oct 30 20:23:16.424579 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:16.424548    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-65f1b6\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:16.437815 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:16.437780    4519 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7059-65f1b6\" is forbidden: User \"system:node:ip-10-0-133-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.ec2.internal' and this object" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" pod="opendatahub/test-curl-7059-65f1b6"
Oct 30 20:23:17.614035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:17.613999    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad99fd9-5bde-4ec8-8d68-74e336bdfe3d" path="/var/lib/kubelet/pods/bad99fd9-5bde-4ec8-8d68-74e336bdfe3d/volumes"
Oct 30 20:23:18.126691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.126657    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"]
Oct 30 20:23:18.127560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.127527    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1a502c5e-7d1a-4489-9f51-52d525a03b94" containerName="revoke-keys"
Oct 30 20:23:18.127560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.127548    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a502c5e-7d1a-4489-9f51-52d525a03b94" containerName="revoke-keys"
Oct 30 20:23:18.127741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.127699    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a502c5e-7d1a-4489-9f51-52d525a03b94" containerName="revoke-keys"
Oct 30 20:23:18.132866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.132844    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.151829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.151798    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"]
Oct 30 20:23:18.155590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.155537    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:18.156757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.156733    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:18.176290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.176259    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.176484 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.176441    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.176565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.176516    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2r9\" (UniqueName: \"kubernetes.io/projected/c5455daa-b677-4e82-8663-16c4ccaa3143-kube-api-access-hv2r9\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.277768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.277726    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.277985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.277857    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.277985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.277895    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2r9\" (UniqueName: \"kubernetes.io/projected/c5455daa-b677-4e82-8663-16c4ccaa3143-kube-api-access-hv2r9\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.278597 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.278575    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.280424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.280396    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.297315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.297290    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2r9\" (UniqueName: \"kubernetes.io/projected/c5455daa-b677-4e82-8663-16c4ccaa3143-kube-api-access-hv2r9\") pod \"maas-api-revoke-sub-da205a19-bzbp7\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.448503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.448447    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:18.764049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:18.764010    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"]
Oct 30 20:23:18.765134 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:18.765094    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5455daa_b677_4e82_8663_16c4ccaa3143.slice/crio-7c8e1bbd3780785ca9b479e0ccdfe86eb0756ff6091e8e2cc72b8ce5dfbf9d12 WatchSource:0}: Error finding container 7c8e1bbd3780785ca9b479e0ccdfe86eb0756ff6091e8e2cc72b8ce5dfbf9d12: Status 404 returned error can't find the container with id 7c8e1bbd3780785ca9b479e0ccdfe86eb0756ff6091e8e2cc72b8ce5dfbf9d12
Oct 30 20:23:19.419448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:19.419413    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7" event={"ID":"c5455daa-b677-4e82-8663-16c4ccaa3143","Type":"ContainerStarted","Data":"89becff8257e45b6f6b6684b265d34cfa0e38e54359a5951918a6a5241b71895"}
Oct 30 20:23:19.419645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:19.419501    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7" event={"ID":"c5455daa-b677-4e82-8663-16c4ccaa3143","Type":"ContainerStarted","Data":"7c8e1bbd3780785ca9b479e0ccdfe86eb0756ff6091e8e2cc72b8ce5dfbf9d12"}
Oct 30 20:23:19.462914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:19.462845    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7" podStartSLOduration=1.462825741 podStartE2EDuration="1.462825741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:19.460158544 +0000 UTC m=+2923.960862271" watchObservedRunningTime="2026-10-30 20:23:19.462825741 +0000 UTC m=+2923.963529466"
Oct 30 20:23:20.428509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:20.428453    4519 generic.go:356] "Generic (PLEG): container finished" podID="c5455daa-b677-4e82-8663-16c4ccaa3143" containerID="89becff8257e45b6f6b6684b265d34cfa0e38e54359a5951918a6a5241b71895" exitCode=0
Oct 30 20:23:20.428509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:20.428497    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7" event={"ID":"c5455daa-b677-4e82-8663-16c4ccaa3143","Type":"ContainerDied","Data":"89becff8257e45b6f6b6684b265d34cfa0e38e54359a5951918a6a5241b71895"}
Oct 30 20:23:21.621543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.621514    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:21.752077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.751986    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-xks-ca\") pod \"c5455daa-b677-4e82-8663-16c4ccaa3143\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") "
Oct 30 20:23:21.752250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.752095    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-openshift-ca\") pod \"c5455daa-b677-4e82-8663-16c4ccaa3143\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") "
Oct 30 20:23:21.752250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.752200    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5455daa-b677-4e82-8663-16c4ccaa3143-kube-api-access-hv2r9\" (UniqueName: \"kubernetes.io/projected/c5455daa-b677-4e82-8663-16c4ccaa3143-kube-api-access-hv2r9\") pod \"c5455daa-b677-4e82-8663-16c4ccaa3143\" (UID: \"c5455daa-b677-4e82-8663-16c4ccaa3143\") "
Oct 30 20:23:21.752931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.752549    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-openshift-ca" pod "c5455daa-b677-4e82-8663-16c4ccaa3143" (UID: "c5455daa-b677-4e82-8663-16c4ccaa3143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:21.752931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.752888    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:21.754630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.754603    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5455daa-b677-4e82-8663-16c4ccaa3143-kube-api-access-hv2r9" pod "c5455daa-b677-4e82-8663-16c4ccaa3143" (UID: "c5455daa-b677-4e82-8663-16c4ccaa3143"). InnerVolumeSpecName "kube-api-access-hv2r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:21.754759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.754735    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-xks-ca" pod "c5455daa-b677-4e82-8663-16c4ccaa3143" (UID: "c5455daa-b677-4e82-8663-16c4ccaa3143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:21.854980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.854933    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5455daa-b677-4e82-8663-16c4ccaa3143-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:21.854980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:21.854977    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2r9\" (UniqueName: \"kubernetes.io/projected/c5455daa-b677-4e82-8663-16c4ccaa3143-kube-api-access-hv2r9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:22.444939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:22.444901    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"
Oct 30 20:23:22.444939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:22.444931    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7" event={"ID":"c5455daa-b677-4e82-8663-16c4ccaa3143","Type":"ContainerDied","Data":"7c8e1bbd3780785ca9b479e0ccdfe86eb0756ff6091e8e2cc72b8ce5dfbf9d12"}
Oct 30 20:23:22.445250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:22.444986    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8e1bbd3780785ca9b479e0ccdfe86eb0756ff6091e8e2cc72b8ce5dfbf9d12"
Oct 30 20:23:23.154994 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.154951    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"]
Oct 30 20:23:23.155952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.155928    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c5455daa-b677-4e82-8663-16c4ccaa3143" containerName="revoke-keys"
Oct 30 20:23:23.156005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.155955    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5455daa-b677-4e82-8663-16c4ccaa3143" containerName="revoke-keys"
Oct 30 20:23:23.156147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.156135    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5455daa-b677-4e82-8663-16c4ccaa3143" containerName="revoke-keys"
Oct 30 20:23:23.160607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.160582    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.178539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.178503    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"]
Oct 30 20:23:23.184123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.184095    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:23.184285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.184226    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:23.207643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.207601    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"]
Oct 30 20:23:23.212886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.212851    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.235577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.235546    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"]
Oct 30 20:23:23.274264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.274228    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.274522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.274425    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.274522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.274455    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwss\" (UniqueName: \"kubernetes.io/projected/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-kube-api-access-lnwss\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.276061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.276031    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5c2e32-openshift-default-dockercfg-6hjx8\""
Oct 30 20:23:23.375743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.375706    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-token\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.375961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.375760    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.375961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.375855    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f20780-1c13-41d8-a471-9ee518ce0acd-istiod-ca-cert\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.375961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.375952    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.376161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.375989    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-credential-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.376161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376020    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-envoy\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.376161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376142    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.376338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376171    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwss\" (UniqueName: \"kubernetes.io/projected/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-kube-api-access-lnwss\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.376338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376203    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-certs\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.376338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376235    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-podinfo\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.376338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376253    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqvv\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-kube-api-access-whqvv\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.376338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376288    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-data\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.376666 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.376647    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.378655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.378634    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.395798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.395769    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwss\" (UniqueName: \"kubernetes.io/projected/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-kube-api-access-lnwss\") pod \"maas-api-revoke-sub-d33e7d88-lldgr\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.477662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.477630    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:23.478204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478176    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-certs\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478236    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-podinfo\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478264    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqvv\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-kube-api-access-whqvv\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478313    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-data\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-token\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f20780-1c13-41d8-a471-9ee518ce0acd-istiod-ca-cert\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478633    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-certs\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478658    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478698    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-data\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478710    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-credential-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.478950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.478778    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-envoy\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.479251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.479231    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f20780-1c13-41d8-a471-9ee518ce0acd-istiod-ca-cert\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.479575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.479555    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-credential-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.479671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.479574    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.480901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.480882    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-podinfo\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.481546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.481528    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-envoy\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.497808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.497785    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqvv\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-kube-api-access-whqvv\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.497931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.497907    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-token\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.533976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.533928    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:23.784837 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:23.784793    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e73240e_bbb2_40fa_87c1_eeb47ee496b5.slice/crio-9ba6df3145d25f8f703f024d221ebfdb73dced7ff35486c34b19004054b1d051 WatchSource:0}: Error finding container 9ba6df3145d25f8f703f024d221ebfdb73dced7ff35486c34b19004054b1d051: Status 404 returned error can't find the container with id 9ba6df3145d25f8f703f024d221ebfdb73dced7ff35486c34b19004054b1d051
Oct 30 20:23:23.786574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.786547    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"]
Oct 30 20:23:23.876631 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:23.876602    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f20780_1c13_41d8_a471_9ee518ce0acd.slice/crio-fbf428cd5f8f7d7dd15e04e0e3fe3e7c685993b83cc46f4df26f774704cce3a7 WatchSource:0}: Error finding container fbf428cd5f8f7d7dd15e04e0e3fe3e7c685993b83cc46f4df26f774704cce3a7: Status 404 returned error can't find the container with id fbf428cd5f8f7d7dd15e04e0e3fe3e7c685993b83cc46f4df26f774704cce3a7
Oct 30 20:23:23.878199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.878171    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"]
Oct 30 20:23:23.879795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.879761    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:23:23.879866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.879826    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:23:23.879866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:23.879860    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:23:24.462800 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.462759    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" event={"ID":"83f20780-1c13-41d8-a471-9ee518ce0acd","Type":"ContainerStarted","Data":"35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c"}
Oct 30 20:23:24.463272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.462819    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" event={"ID":"83f20780-1c13-41d8-a471-9ee518ce0acd","Type":"ContainerStarted","Data":"fbf428cd5f8f7d7dd15e04e0e3fe3e7c685993b83cc46f4df26f774704cce3a7"}
Oct 30 20:23:24.464904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.464858    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr" event={"ID":"4e73240e-bbb2-40fa-87c1-eeb47ee496b5","Type":"ContainerStarted","Data":"ad64e03f618835568bfbb440b8c1cc217fa2de6fcc139e7424f921f877fda3ae"}
Oct 30 20:23:24.465052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.464919    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr" event={"ID":"4e73240e-bbb2-40fa-87c1-eeb47ee496b5","Type":"ContainerStarted","Data":"9ba6df3145d25f8f703f024d221ebfdb73dced7ff35486c34b19004054b1d051"}
Oct 30 20:23:24.535265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.535230    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:24.537340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.537309    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 20:23:24.537517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.537367    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 20:23:24.541134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.541089    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podStartSLOduration=1.5410775239999999 podStartE2EDuration="1.541077524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:24.528898032 +0000 UTC m=+2929.029601785" watchObservedRunningTime="2026-10-30 20:23:24.541077524 +0000 UTC m=+2929.041781248"
Oct 30 20:23:24.580296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:24.580182    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr" podStartSLOduration=1.580162597 podStartE2EDuration="1.580162597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:24.575898506 +0000 UTC m=+2929.076602230" watchObservedRunningTime="2026-10-30 20:23:24.580162597 +0000 UTC m=+2929.080866326"
Oct 30 20:23:25.485799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:25.485762    4519 generic.go:356] "Generic (PLEG): container finished" podID="4e73240e-bbb2-40fa-87c1-eeb47ee496b5" containerID="ad64e03f618835568bfbb440b8c1cc217fa2de6fcc139e7424f921f877fda3ae" exitCode=0
Oct 30 20:23:25.486201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:25.485851    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr" event={"ID":"4e73240e-bbb2-40fa-87c1-eeb47ee496b5","Type":"ContainerDied","Data":"ad64e03f618835568bfbb440b8c1cc217fa2de6fcc139e7424f921f877fda3ae"}
Oct 30 20:23:25.535253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:25.535208    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 20:23:25.535479 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:25.535273    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 20:23:26.535107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.535062    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 20:23:26.535655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.535126    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 20:23:26.722452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.722423    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:26.731314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.731275    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-openshift-ca\") pod \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") "
Oct 30 20:23:26.731564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.731352    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-kube-api-access-lnwss\" (UniqueName: \"kubernetes.io/projected/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-kube-api-access-lnwss\") pod \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") "
Oct 30 20:23:26.731564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.731428    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-xks-ca\") pod \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\" (UID: \"4e73240e-bbb2-40fa-87c1-eeb47ee496b5\") "
Oct 30 20:23:26.731951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.731921    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-openshift-ca" pod "4e73240e-bbb2-40fa-87c1-eeb47ee496b5" (UID: "4e73240e-bbb2-40fa-87c1-eeb47ee496b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.732169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.732142    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:26.733786 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.733764    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-xks-ca" pod "4e73240e-bbb2-40fa-87c1-eeb47ee496b5" (UID: "4e73240e-bbb2-40fa-87c1-eeb47ee496b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:26.733874 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.733823    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-kube-api-access-lnwss" pod "4e73240e-bbb2-40fa-87c1-eeb47ee496b5" (UID: "4e73240e-bbb2-40fa-87c1-eeb47ee496b5"). InnerVolumeSpecName "kube-api-access-lnwss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.833179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.833089    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwss\" (UniqueName: \"kubernetes.io/projected/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-kube-api-access-lnwss\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:26.833374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:26.833361    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e73240e-bbb2-40fa-87c1-eeb47ee496b5-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:27.591138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:27.588623    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 20:23:27.591138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:27.588683    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 20:23:27.598589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:27.598547    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr" event={"ID":"4e73240e-bbb2-40fa-87c1-eeb47ee496b5","Type":"ContainerDied","Data":"9ba6df3145d25f8f703f024d221ebfdb73dced7ff35486c34b19004054b1d051"}
Oct 30 20:23:27.598757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:27.598609    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ba6df3145d25f8f703f024d221ebfdb73dced7ff35486c34b19004054b1d051"
Oct 30 20:23:27.598757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:27.598609    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"
Oct 30 20:23:28.535076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:28.535036    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 20:23:28.535311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:28.535095    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 20:23:29.025571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.025529    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"]
Oct 30 20:23:29.026259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.026240    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4e73240e-bbb2-40fa-87c1-eeb47ee496b5" containerName="revoke-keys"
Oct 30 20:23:29.026328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.026260    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e73240e-bbb2-40fa-87c1-eeb47ee496b5" containerName="revoke-keys"
Oct 30 20:23:29.026399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.026390    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e73240e-bbb2-40fa-87c1-eeb47ee496b5" containerName="revoke-keys"
Oct 30 20:23:29.030201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.030183    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.079348 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.069433    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:29.079348 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.069592    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:29.093485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.093434    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"]
Oct 30 20:23:29.132006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.131977    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6w9\" (UniqueName: \"kubernetes.io/projected/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-kube-api-access-rh6w9\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.132457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.132440    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.132765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.132749    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.235892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.235850    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.236169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.236139    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.236390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.236367    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6w9\" (UniqueName: \"kubernetes.io/projected/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-kube-api-access-rh6w9\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.237107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.237082    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.238615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.238592    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.278798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.278715    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6w9\" (UniqueName: \"kubernetes.io/projected/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-kube-api-access-rh6w9\") pod \"maas-api-revoke-sub-eebaeeeb-sdbxr\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.345806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.345770    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:29.535363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.535265    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 20:23:29.535363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.535333    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 20:23:29.693539 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:29.693503    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4c3c22_0d2f_42a3_ac04_d3e585b57154.slice/crio-9445e0272c17fb73b3eb21aca29ac947cda2f08e00978e3cc9a2343cd4dade14 WatchSource:0}: Error finding container 9445e0272c17fb73b3eb21aca29ac947cda2f08e00978e3cc9a2343cd4dade14: Status 404 returned error can't find the container with id 9445e0272c17fb73b3eb21aca29ac947cda2f08e00978e3cc9a2343cd4dade14
Oct 30 20:23:29.694423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:29.694397    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"]
Oct 30 20:23:30.534547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:30.534507    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 20:23:30.535058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:30.534587    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 20:23:30.628492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:30.627778    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr" event={"ID":"eb4c3c22-0d2f-42a3-ac04-d3e585b57154","Type":"ContainerStarted","Data":"f63df2993c5e459a268cad947b934dcc05de5a039c340e2665696d649a92f722"}
Oct 30 20:23:30.628492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:30.627832    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr" event={"ID":"eb4c3c22-0d2f-42a3-ac04-d3e585b57154","Type":"ContainerStarted","Data":"9445e0272c17fb73b3eb21aca29ac947cda2f08e00978e3cc9a2343cd4dade14"}
Oct 30 20:23:30.668202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:30.668134    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr" podStartSLOduration=2.668115509 podStartE2EDuration="2.668115509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:30.667533921 +0000 UTC m=+2935.168237645" watchObservedRunningTime="2026-10-30 20:23:30.668115509 +0000 UTC m=+2935.168819234"
Oct 30 20:23:31.539801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:31.539766    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:31.540328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:31.540246    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:31.541003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:31.540977    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:23:31.647100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:31.647057    4519 generic.go:356] "Generic (PLEG): container finished" podID="eb4c3c22-0d2f-42a3-ac04-d3e585b57154" containerID="f63df2993c5e459a268cad947b934dcc05de5a039c340e2665696d649a92f722" exitCode=0
Oct 30 20:23:31.647459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:31.647342    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr" event={"ID":"eb4c3c22-0d2f-42a3-ac04-d3e585b57154","Type":"ContainerDied","Data":"f63df2993c5e459a268cad947b934dcc05de5a039c340e2665696d649a92f722"}
Oct 30 20:23:32.253291 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:32.253252    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:23:32.844559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.844530    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:32.912917 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.912888    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-kube-api-access-rh6w9\" (UniqueName: \"kubernetes.io/projected/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-kube-api-access-rh6w9\") pod \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") "
Oct 30 20:23:32.913109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.913044    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-xks-ca\") pod \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") "
Oct 30 20:23:32.913109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.913083    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-openshift-ca\") pod \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\" (UID: \"eb4c3c22-0d2f-42a3-ac04-d3e585b57154\") "
Oct 30 20:23:32.913755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.913701    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-openshift-ca" pod "eb4c3c22-0d2f-42a3-ac04-d3e585b57154" (UID: "eb4c3c22-0d2f-42a3-ac04-d3e585b57154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:32.913983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.913951    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:32.915525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.915483    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-xks-ca" pod "eb4c3c22-0d2f-42a3-ac04-d3e585b57154" (UID: "eb4c3c22-0d2f-42a3-ac04-d3e585b57154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:32.915639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:32.915596    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-kube-api-access-rh6w9" pod "eb4c3c22-0d2f-42a3-ac04-d3e585b57154" (UID: "eb4c3c22-0d2f-42a3-ac04-d3e585b57154"). InnerVolumeSpecName "kube-api-access-rh6w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:33.016179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:33.016130    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:33.016179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:33.016179    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6w9\" (UniqueName: \"kubernetes.io/projected/eb4c3c22-0d2f-42a3-ac04-d3e585b57154-kube-api-access-rh6w9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:33.668523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:33.668451    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr" event={"ID":"eb4c3c22-0d2f-42a3-ac04-d3e585b57154","Type":"ContainerDied","Data":"9445e0272c17fb73b3eb21aca29ac947cda2f08e00978e3cc9a2343cd4dade14"}
Oct 30 20:23:33.668781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:33.668522    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"
Oct 30 20:23:33.668781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:33.668546    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9445e0272c17fb73b3eb21aca29ac947cda2f08e00978e3cc9a2343cd4dade14"
Oct 30 20:23:34.146183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.146142    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"]
Oct 30 20:23:34.147057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.147040    4519 cpu_manager.go:414] "containerMap: removing container" podUID="eb4c3c22-0d2f-42a3-ac04-d3e585b57154" containerName="revoke-keys"
Oct 30 20:23:34.147128 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.147060    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4c3c22-0d2f-42a3-ac04-d3e585b57154" containerName="revoke-keys"
Oct 30 20:23:34.147239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.147227    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4c3c22-0d2f-42a3-ac04-d3e585b57154" containerName="revoke-keys"
Oct 30 20:23:34.151425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.151400    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.169507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.169483    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:34.181686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.181665    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:34.209456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.209425    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"]
Oct 30 20:23:34.244289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.244244    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw29v\" (UniqueName: \"kubernetes.io/projected/b1daca35-8117-4cd3-83e6-34b2edf18c2f-kube-api-access-sw29v\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.244491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.244298    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.244491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.244397    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.346125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.346086    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw29v\" (UniqueName: \"kubernetes.io/projected/b1daca35-8117-4cd3-83e6-34b2edf18c2f-kube-api-access-sw29v\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.346313 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.346150    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.346313 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.346186    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.346973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.346947    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.348691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.348662    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.370413 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.370386    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw29v\" (UniqueName: \"kubernetes.io/projected/b1daca35-8117-4cd3-83e6-34b2edf18c2f-kube-api-access-sw29v\") pod \"maas-api-revoke-sub-da59d6e3-hkxmw\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.465728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.465694    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:34.772021 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:34.771986    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-443408b46512111af4feb778484c5143bfbc381e15157bd724713096b2c6c0ba WatchSource:0}: Error finding container 443408b46512111af4feb778484c5143bfbc381e15157bd724713096b2c6c0ba: Status 404 returned error can't find the container with id 443408b46512111af4feb778484c5143bfbc381e15157bd724713096b2c6c0ba
Oct 30 20:23:34.774271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:34.774245    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"]
Oct 30 20:23:35.686175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:35.686127    4519 generic.go:356] "Generic (PLEG): container finished" podID="b1daca35-8117-4cd3-83e6-34b2edf18c2f" containerID="14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779" exitCode=0
Oct 30 20:23:35.686617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:35.686363    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw" event={"ID":"b1daca35-8117-4cd3-83e6-34b2edf18c2f","Type":"ContainerDied","Data":"14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779"}
Oct 30 20:23:35.686617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:35.686420    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw" event={"ID":"b1daca35-8117-4cd3-83e6-34b2edf18c2f","Type":"ContainerStarted","Data":"443408b46512111af4feb778484c5143bfbc381e15157bd724713096b2c6c0ba"}
Oct 30 20:23:36.876785 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.876753    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:36.994022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.993929    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-openshift-ca\") pod \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") "
Oct 30 20:23:36.994022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.993982    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-xks-ca\") pod \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") "
Oct 30 20:23:36.994261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.994137    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1daca35-8117-4cd3-83e6-34b2edf18c2f-kube-api-access-sw29v\" (UniqueName: \"kubernetes.io/projected/b1daca35-8117-4cd3-83e6-34b2edf18c2f-kube-api-access-sw29v\") pod \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\" (UID: \"b1daca35-8117-4cd3-83e6-34b2edf18c2f\") "
Oct 30 20:23:36.994411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.994378    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-openshift-ca" pod "b1daca35-8117-4cd3-83e6-34b2edf18c2f" (UID: "b1daca35-8117-4cd3-83e6-34b2edf18c2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:36.994885 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.994858    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:36.996477 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.996435    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-xks-ca" pod "b1daca35-8117-4cd3-83e6-34b2edf18c2f" (UID: "b1daca35-8117-4cd3-83e6-34b2edf18c2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:36.996595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:36.996448    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1daca35-8117-4cd3-83e6-34b2edf18c2f-kube-api-access-sw29v" pod "b1daca35-8117-4cd3-83e6-34b2edf18c2f" (UID: "b1daca35-8117-4cd3-83e6-34b2edf18c2f"). InnerVolumeSpecName "kube-api-access-sw29v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:37.097103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:37.097069    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1daca35-8117-4cd3-83e6-34b2edf18c2f-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:37.097103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:37.097100    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw29v\" (UniqueName: \"kubernetes.io/projected/b1daca35-8117-4cd3-83e6-34b2edf18c2f-kube-api-access-sw29v\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:37.707145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:37.707099    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"
Oct 30 20:23:37.707304 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:37.707132    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw" event={"ID":"b1daca35-8117-4cd3-83e6-34b2edf18c2f","Type":"ContainerDied","Data":"443408b46512111af4feb778484c5143bfbc381e15157bd724713096b2c6c0ba"}
Oct 30 20:23:37.707304 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:37.707203    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="443408b46512111af4feb778484c5143bfbc381e15157bd724713096b2c6c0ba"
Oct 30 20:23:39.795693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.795652    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"]
Oct 30 20:23:39.796315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.796299    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b1daca35-8117-4cd3-83e6-34b2edf18c2f" containerName="revoke-keys"
Oct 30 20:23:39.796392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.796316    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1daca35-8117-4cd3-83e6-34b2edf18c2f" containerName="revoke-keys"
Oct 30 20:23:39.796449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.796416    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1daca35-8117-4cd3-83e6-34b2edf18c2f" containerName="revoke-keys"
Oct 30 20:23:39.800432 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.800412    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:39.821297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.821269    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"]
Oct 30 20:23:39.828505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.827525    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:39.828505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.827609    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:39.946837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.946794    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8d42\" (UniqueName: \"kubernetes.io/projected/d9a6abac-bec4-48fb-a537-14fd3fec31cc-kube-api-access-s8d42\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:39.947035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.947019    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:39.947082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:39.947049    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.048508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.048366    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8d42\" (UniqueName: \"kubernetes.io/projected/d9a6abac-bec4-48fb-a537-14fd3fec31cc-kube-api-access-s8d42\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.048708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.048571    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.048708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.048609    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.049270 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.049245    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.054935 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.054901    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.073408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.072240    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8d42\" (UniqueName: \"kubernetes.io/projected/d9a6abac-bec4-48fb-a537-14fd3fec31cc-kube-api-access-s8d42\") pod \"maas-api-revoke-sub-8733451b-r94fz\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.114511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.114431    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:40.436321 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:40.436284    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a6abac_bec4_48fb_a537_14fd3fec31cc.slice/crio-2b304713c9a9df2a67da37a4d57c2512640a93211aad048fe448fab9c7fcd295 WatchSource:0}: Error finding container 2b304713c9a9df2a67da37a4d57c2512640a93211aad048fe448fab9c7fcd295: Status 404 returned error can't find the container with id 2b304713c9a9df2a67da37a4d57c2512640a93211aad048fe448fab9c7fcd295
Oct 30 20:23:40.436950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.436927    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"]
Oct 30 20:23:40.767342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:40.767303    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz" event={"ID":"d9a6abac-bec4-48fb-a537-14fd3fec31cc","Type":"ContainerStarted","Data":"2b304713c9a9df2a67da37a4d57c2512640a93211aad048fe448fab9c7fcd295"}
Oct 30 20:23:41.312420 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:41.312381    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:23:41.778371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:41.778331    4519 generic.go:356] "Generic (PLEG): container finished" podID="d9a6abac-bec4-48fb-a537-14fd3fec31cc" containerID="305848aec4ede14d0790c35b7b26082e2eb3fd7891255b267a6ccd2f69c81be1" exitCode=0
Oct 30 20:23:41.778577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:41.778420    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz" event={"ID":"d9a6abac-bec4-48fb-a537-14fd3fec31cc","Type":"ContainerDied","Data":"305848aec4ede14d0790c35b7b26082e2eb3fd7891255b267a6ccd2f69c81be1"}
Oct 30 20:23:41.903549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:41.895773    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"]
Oct 30 20:23:41.903549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:41.901337    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:41.925103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:41.925065    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"]
Oct 30 20:23:41.925724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:41.925698    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f35cc1-openshift-default-dockercfg-rw9cm\""
Oct 30 20:23:42.012444 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.012404    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-socket\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.012444 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.012449    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmt4\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-kube-api-access-zmmt4\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.013022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.012638    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-envoy\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.013022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.012688    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-token\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.013022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.012729    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-data\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.013022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.012855    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-credential-socket\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.013022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.012958    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-certs\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.013022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.013001    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-podinfo\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.013243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.013145    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6155870-2e23-4484-8c1d-1b34e19e2bca-istiod-ca-cert\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115079    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6155870-2e23-4484-8c1d-1b34e19e2bca-istiod-ca-cert\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115144    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-socket\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115177    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmt4\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-kube-api-access-zmmt4\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115239    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-envoy\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115267    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-token\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115302    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-data\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115394    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-credential-socket\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115529    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-certs\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.116703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.115568    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-podinfo\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.121870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.121832    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6155870-2e23-4484-8c1d-1b34e19e2bca-istiod-ca-cert\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.123301 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.123270    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-data\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.124236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.124205    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-credential-socket\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.124524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.124502    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-certs\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.125325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.125300    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-socket\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.128188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.128164    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-podinfo\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.136983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.136781    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-envoy\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.157372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.157336    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmt4\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-kube-api-access-zmmt4\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.158036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.158009    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-token\") pod \"e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") " pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.228849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.228806    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:42.872300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.871649    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"]
Oct 30 20:23:42.909236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.909184    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:23:42.909387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.909326    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:23:42.909387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:42.909369    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:23:43.076402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.071746    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:43.097554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.096538    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a6abac-bec4-48fb-a537-14fd3fec31cc-kube-api-access-s8d42\" (UniqueName: \"kubernetes.io/projected/d9a6abac-bec4-48fb-a537-14fd3fec31cc-kube-api-access-s8d42\") pod \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") "
Oct 30 20:23:43.097554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.096782    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-openshift-ca\") pod \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") "
Oct 30 20:23:43.097554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.097145    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-openshift-ca" pod "d9a6abac-bec4-48fb-a537-14fd3fec31cc" (UID: "d9a6abac-bec4-48fb-a537-14fd3fec31cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:43.097554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.097219    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-xks-ca\") pod \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\" (UID: \"d9a6abac-bec4-48fb-a537-14fd3fec31cc\") "
Oct 30 20:23:43.099942 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.098318    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:43.104534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.102804    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a6abac-bec4-48fb-a537-14fd3fec31cc-kube-api-access-s8d42" pod "d9a6abac-bec4-48fb-a537-14fd3fec31cc" (UID: "d9a6abac-bec4-48fb-a537-14fd3fec31cc"). InnerVolumeSpecName "kube-api-access-s8d42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:43.104534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.103072    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-xks-ca" pod "d9a6abac-bec4-48fb-a537-14fd3fec31cc" (UID: "d9a6abac-bec4-48fb-a537-14fd3fec31cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:43.201330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.201214    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8d42\" (UniqueName: \"kubernetes.io/projected/d9a6abac-bec4-48fb-a537-14fd3fec31cc-kube-api-access-s8d42\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:43.201330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.201256    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a6abac-bec4-48fb-a537-14fd3fec31cc-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:43.645858 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:43.645814    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:23:43.845694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.845650    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz" event={"ID":"d9a6abac-bec4-48fb-a537-14fd3fec31cc","Type":"ContainerDied","Data":"2b304713c9a9df2a67da37a4d57c2512640a93211aad048fe448fab9c7fcd295"}
Oct 30 20:23:43.845895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.845752    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b304713c9a9df2a67da37a4d57c2512640a93211aad048fe448fab9c7fcd295"
Oct 30 20:23:43.845895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.845665    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"
Oct 30 20:23:43.847891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.847851    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" event={"ID":"c6155870-2e23-4484-8c1d-1b34e19e2bca","Type":"ContainerStarted","Data":"0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee"}
Oct 30 20:23:43.848101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.847913    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" event={"ID":"c6155870-2e23-4484-8c1d-1b34e19e2bca","Type":"ContainerStarted","Data":"8df88b10a7c3b194782113c7c8a9c5ca1abbfd14bb644b1d83a4dbac3e733862"}
Oct 30 20:23:43.906360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:43.906233    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podStartSLOduration=2.90622007 podStartE2EDuration="2.90622007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:43.903907025 +0000 UTC m=+2948.404610749" watchObservedRunningTime="2026-10-30 20:23:43.90622007 +0000 UTC m=+2948.406923793"
Oct 30 20:23:44.230375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.230331    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:44.237673 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.237643    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:44.702788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.702755    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"]
Oct 30 20:23:44.703366 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.703353    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d9a6abac-bec4-48fb-a537-14fd3fec31cc" containerName="revoke-keys"
Oct 30 20:23:44.703420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.703368    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a6abac-bec4-48fb-a537-14fd3fec31cc" containerName="revoke-keys"
Oct 30 20:23:44.703515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.703505    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a6abac-bec4-48fb-a537-14fd3fec31cc" containerName="revoke-keys"
Oct 30 20:23:44.706834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.706816    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.725257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.725231    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:44.725257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.725231    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:23:44.727196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.727154    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlk75\" (UniqueName: \"kubernetes.io/projected/c56c44d3-bcc0-4e39-b64d-db292a39e418-kube-api-access-nlk75\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.727425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.727398    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.727727 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.727704    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.729027 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.729005    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"]
Oct 30 20:23:44.767362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.767278    4519 scope.go:122] "RemoveContainer" containerID="fdbe96550442cfc40adda5248941239edfdf4d06ca1c83450daa495042e1d00b"
Oct 30 20:23:44.801132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.801101    4519 scope.go:122] "RemoveContainer" containerID="02380a49c2a6ee9e98833ca9c6fa7c8bd66b8b8cded639897c540a4047d3584c"
Oct 30 20:23:44.828942 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.828916    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlk75\" (UniqueName: \"kubernetes.io/projected/c56c44d3-bcc0-4e39-b64d-db292a39e418-kube-api-access-nlk75\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.829070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.829056    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.829166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.829155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.829855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.829831    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.831768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.831742    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.849089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.849056    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlk75\" (UniqueName: \"kubernetes.io/projected/c56c44d3-bcc0-4e39-b64d-db292a39e418-kube-api-access-nlk75\") pod \"maas-api-revoke-sub-08d3c945-qjtv7\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:44.860057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.860031    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:44.860971 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:44.860949    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:23:45.027261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:45.025359    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:45.340616 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:23:45.340584    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56c44d3_bcc0_4e39_b64d_db292a39e418.slice/crio-64b788b7a90f44c4ebb1f638e937f2a3985ec3c3d2ec650d319fc1d5c7ae2553 WatchSource:0}: Error finding container 64b788b7a90f44c4ebb1f638e937f2a3985ec3c3d2ec650d319fc1d5c7ae2553: Status 404 returned error can't find the container with id 64b788b7a90f44c4ebb1f638e937f2a3985ec3c3d2ec650d319fc1d5c7ae2553
Oct 30 20:23:45.341909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:45.341882    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"]
Oct 30 20:23:45.870802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:45.870758    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7" event={"ID":"c56c44d3-bcc0-4e39-b64d-db292a39e418","Type":"ContainerStarted","Data":"5ba5a954db08cc383797f1ae9d54b5d447dd8f1269a6bb25ef7c94d4f277a8a1"}
Oct 30 20:23:45.870988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:45.870823    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7" event={"ID":"c56c44d3-bcc0-4e39-b64d-db292a39e418","Type":"ContainerStarted","Data":"64b788b7a90f44c4ebb1f638e937f2a3985ec3c3d2ec650d319fc1d5c7ae2553"}
Oct 30 20:23:45.911755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:45.911678    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7" podStartSLOduration=1.911659121 podStartE2EDuration="1.911659121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:45.909999779 +0000 UTC m=+2950.410703502" watchObservedRunningTime="2026-10-30 20:23:45.911659121 +0000 UTC m=+2950.412362845"
Oct 30 20:23:46.882667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:46.882626    4519 generic.go:356] "Generic (PLEG): container finished" podID="c56c44d3-bcc0-4e39-b64d-db292a39e418" containerID="5ba5a954db08cc383797f1ae9d54b5d447dd8f1269a6bb25ef7c94d4f277a8a1" exitCode=0
Oct 30 20:23:46.883109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:46.882708    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7" event={"ID":"c56c44d3-bcc0-4e39-b64d-db292a39e418","Type":"ContainerDied","Data":"5ba5a954db08cc383797f1ae9d54b5d447dd8f1269a6bb25ef7c94d4f277a8a1"}
Oct 30 20:23:48.135332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.135300    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:48.208391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.208361    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-openshift-ca\") pod \"c56c44d3-bcc0-4e39-b64d-db292a39e418\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") "
Oct 30 20:23:48.208391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.208403    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56c44d3-bcc0-4e39-b64d-db292a39e418-kube-api-access-nlk75\" (UniqueName: \"kubernetes.io/projected/c56c44d3-bcc0-4e39-b64d-db292a39e418-kube-api-access-nlk75\") pod \"c56c44d3-bcc0-4e39-b64d-db292a39e418\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") "
Oct 30 20:23:48.208703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.208687    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-xks-ca\") pod \"c56c44d3-bcc0-4e39-b64d-db292a39e418\" (UID: \"c56c44d3-bcc0-4e39-b64d-db292a39e418\") "
Oct 30 20:23:48.209170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.209137    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-openshift-ca" pod "c56c44d3-bcc0-4e39-b64d-db292a39e418" (UID: "c56c44d3-bcc0-4e39-b64d-db292a39e418"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:48.211214 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.211185    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56c44d3-bcc0-4e39-b64d-db292a39e418-kube-api-access-nlk75" pod "c56c44d3-bcc0-4e39-b64d-db292a39e418" (UID: "c56c44d3-bcc0-4e39-b64d-db292a39e418"). InnerVolumeSpecName "kube-api-access-nlk75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:48.211661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.211638    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-xks-ca" pod "c56c44d3-bcc0-4e39-b64d-db292a39e418" (UID: "c56c44d3-bcc0-4e39-b64d-db292a39e418"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:48.311113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.311073    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:48.311113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.311113    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56c44d3-bcc0-4e39-b64d-db292a39e418-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:48.311394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.311132    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlk75\" (UniqueName: \"kubernetes.io/projected/c56c44d3-bcc0-4e39-b64d-db292a39e418-kube-api-access-nlk75\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:23:48.901113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.901078    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"
Oct 30 20:23:48.901299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.901112    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7" event={"ID":"c56c44d3-bcc0-4e39-b64d-db292a39e418","Type":"ContainerDied","Data":"64b788b7a90f44c4ebb1f638e937f2a3985ec3c3d2ec650d319fc1d5c7ae2553"}
Oct 30 20:23:48.901299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:48.901172    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64b788b7a90f44c4ebb1f638e937f2a3985ec3c3d2ec650d319fc1d5c7ae2553"
Oct 30 20:23:50.705548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:23:50.705507    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"]
Oct 30 20:23:53.756930 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:53.756884    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:23:56.246781 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:23:56.246729    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:01.231932 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:01.231893    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:01.232500 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:01.231972    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:02.223367 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:02.223326    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:24:03.421414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:03.421084    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"]
Oct 30 20:24:03.790547 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:03.787058    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:05.384478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.384415    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"]
Oct 30 20:24:05.386089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.386057    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c56c44d3-bcc0-4e39-b64d-db292a39e418" containerName="revoke-keys"
Oct 30 20:24:05.386267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.386250    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56c44d3-bcc0-4e39-b64d-db292a39e418" containerName="revoke-keys"
Oct 30 20:24:05.392716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.392692    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56c44d3-bcc0-4e39-b64d-db292a39e418" containerName="revoke-keys"
Oct 30 20:24:05.403957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.403921    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"]
Oct 30 20:24:05.404110 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.404042    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.418658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.418632    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:05.430522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.430497    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:24:05.511295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.511264    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.511555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.511538    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.511779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.511750    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkq46\" (UniqueName: \"kubernetes.io/projected/41e3944f-39f9-4d96-b483-9188928feea0-kube-api-access-kkq46\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.613162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.613127    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.613373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.613185    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.613373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.613282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkq46\" (UniqueName: \"kubernetes.io/projected/41e3944f-39f9-4d96-b483-9188928feea0-kube-api-access-kkq46\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.613867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.613843    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.616294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.616264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.633318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.633295    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkq46\" (UniqueName: \"kubernetes.io/projected/41e3944f-39f9-4d96-b483-9188928feea0-kube-api-access-kkq46\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.729232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.729039    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:05.736496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:05.736439    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll" podUID="1fd518d9-2f66-41af-b22b-dc2d1a7be122" containerName="main" containerID="cri-o://13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459" gracePeriod=45
Oct 30 20:24:06.118494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.117498    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"]
Oct 30 20:24:06.184930 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.184566    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:24:06.186186 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:24:06.185268    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e3944f_39f9_4d96_b483_9188928feea0.slice/crio-975b8240b15e3e2ea5fe42e7712881558478ed75812b2f15cf7d45f662566d8d WatchSource:0}: Error finding container 975b8240b15e3e2ea5fe42e7712881558478ed75812b2f15cf7d45f662566d8d: Status 404 returned error can't find the container with id 975b8240b15e3e2ea5fe42e7712881558478ed75812b2f15cf7d45f662566d8d
Oct 30 20:24:06.210402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.210372    4519 generic.go:356] "Generic (PLEG): container finished" podID="1fd518d9-2f66-41af-b22b-dc2d1a7be122" containerID="13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459" exitCode=0
Oct 30 20:24:06.210576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.210502    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll" event={"ID":"1fd518d9-2f66-41af-b22b-dc2d1a7be122","Type":"ContainerDied","Data":"13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459"}
Oct 30 20:24:06.210576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.210541    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll" event={"ID":"1fd518d9-2f66-41af-b22b-dc2d1a7be122","Type":"ContainerDied","Data":"0b78a5138876cebd2cc11d988127dcdc3d16bcd4109aa929b4b62cb75b6b5e81"}
Oct 30 20:24:06.210576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.210567    4519 scope.go:122] "RemoveContainer" containerID="13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459"
Oct 30 20:24:06.210774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.210596    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"
Oct 30 20:24:06.212547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.212514    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t" event={"ID":"41e3944f-39f9-4d96-b483-9188928feea0","Type":"ContainerStarted","Data":"975b8240b15e3e2ea5fe42e7712881558478ed75812b2f15cf7d45f662566d8d"}
Oct 30 20:24:06.223510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.223440    4519 scope.go:122] "RemoveContainer" containerID="13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459"
Oct 30 20:24:06.223983 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:06.223960    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459\": container with ID starting with 13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459 not found: ID does not exist" containerID="13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459"
Oct 30 20:24:06.224068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.223994    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459"} err="failed to get container status \"13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459\": rpc error: code = NotFound desc = could not find container \"13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459\": container with ID starting with 13866ee12e1bf341066dbf519500801fb55aafdb184e05dad2781230a0eb1459 not found: ID does not exist"
Oct 30 20:24:06.342740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.342649    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-home\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-home\") pod \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") "
Oct 30 20:24:06.342918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.342739    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tmp-dir\") pod \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") "
Oct 30 20:24:06.342918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.342776    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd518d9-2f66-41af-b22b-dc2d1a7be122-kube-api-access-5gshs\" (UniqueName: \"kubernetes.io/projected/1fd518d9-2f66-41af-b22b-dc2d1a7be122-kube-api-access-5gshs\") pod \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") "
Oct 30 20:24:06.342918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.342833    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-model-cache\") pod \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") "
Oct 30 20:24:06.343085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.342905    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-home" pod "1fd518d9-2f66-41af-b22b-dc2d1a7be122" (UID: "1fd518d9-2f66-41af-b22b-dc2d1a7be122"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.343085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.342929    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tls-certs\") pod \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") "
Oct 30 20:24:06.343085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.342969    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-dshm\") pod \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\" (UID: \"1fd518d9-2f66-41af-b22b-dc2d1a7be122\") "
Oct 30 20:24:06.343085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.343031    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tmp-dir" pod "1fd518d9-2f66-41af-b22b-dc2d1a7be122" (UID: "1fd518d9-2f66-41af-b22b-dc2d1a7be122"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.343296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.343123    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-model-cache" pod "1fd518d9-2f66-41af-b22b-dc2d1a7be122" (UID: "1fd518d9-2f66-41af-b22b-dc2d1a7be122"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.343696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.343673    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:06.343816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.343698    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:06.343816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.343712    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:06.347984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.347958    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-dshm" pod "1fd518d9-2f66-41af-b22b-dc2d1a7be122" (UID: "1fd518d9-2f66-41af-b22b-dc2d1a7be122"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:06.348113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.348013    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tls-certs" pod "1fd518d9-2f66-41af-b22b-dc2d1a7be122" (UID: "1fd518d9-2f66-41af-b22b-dc2d1a7be122"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:06.348113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.348021    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd518d9-2f66-41af-b22b-dc2d1a7be122-kube-api-access-5gshs" pod "1fd518d9-2f66-41af-b22b-dc2d1a7be122" (UID: "1fd518d9-2f66-41af-b22b-dc2d1a7be122"). InnerVolumeSpecName "kube-api-access-5gshs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:06.445294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.445248    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd518d9-2f66-41af-b22b-dc2d1a7be122-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:06.445294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.445282    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd518d9-2f66-41af-b22b-dc2d1a7be122-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:06.445837 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.445305    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gshs\" (UniqueName: \"kubernetes.io/projected/1fd518d9-2f66-41af-b22b-dc2d1a7be122-kube-api-access-5gshs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:06.559516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.559440    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"]
Oct 30 20:24:06.571322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:06.571291    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll"]
Oct 30 20:24:07.036593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:07.036543    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-5c2e32/sub-test-model-5c2e32-kserve-55d4758c8f-qg9ll" podUID="1fd518d9-2f66-41af-b22b-dc2d1a7be122" containerName="main" probeResult="failure" output="Get \"https://10.133.1.11:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:24:07.225051 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:07.225015    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t" event={"ID":"41e3944f-39f9-4d96-b483-9188928feea0","Type":"ContainerStarted","Data":"da11bc93a4b6632d5f3afd73394aebf31a347f4476cc2f6e62bd2881c839d65f"}
Oct 30 20:24:07.276831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:07.276461    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t" podStartSLOduration=2.276442304 podStartE2EDuration="2.276442304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:07.26607994 +0000 UTC m=+2971.766783677" watchObservedRunningTime="2026-10-30 20:24:07.276442304 +0000 UTC m=+2971.777146032"
Oct 30 20:24:07.615285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:07.615246    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd518d9-2f66-41af-b22b-dc2d1a7be122" path="/var/lib/kubelet/pods/1fd518d9-2f66-41af-b22b-dc2d1a7be122/volumes"
Oct 30 20:24:08.235778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:08.235740    4519 generic.go:356] "Generic (PLEG): container finished" podID="41e3944f-39f9-4d96-b483-9188928feea0" containerID="da11bc93a4b6632d5f3afd73394aebf31a347f4476cc2f6e62bd2881c839d65f" exitCode=0
Oct 30 20:24:08.235977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:08.235795    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t" event={"ID":"41e3944f-39f9-4d96-b483-9188928feea0","Type":"ContainerDied","Data":"da11bc93a4b6632d5f3afd73394aebf31a347f4476cc2f6e62bd2881c839d65f"}
Oct 30 20:24:09.430801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.430769    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:09.596589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.596513    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e3944f-39f9-4d96-b483-9188928feea0-kube-api-access-kkq46\" (UniqueName: \"kubernetes.io/projected/41e3944f-39f9-4d96-b483-9188928feea0-kube-api-access-kkq46\") pod \"41e3944f-39f9-4d96-b483-9188928feea0\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") "
Oct 30 20:24:09.596767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.596629    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-openshift-ca\") pod \"41e3944f-39f9-4d96-b483-9188928feea0\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") "
Oct 30 20:24:09.596872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.596848    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-xks-ca\") pod \"41e3944f-39f9-4d96-b483-9188928feea0\" (UID: \"41e3944f-39f9-4d96-b483-9188928feea0\") "
Oct 30 20:24:09.597098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.597066    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-openshift-ca" pod "41e3944f-39f9-4d96-b483-9188928feea0" (UID: "41e3944f-39f9-4d96-b483-9188928feea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:09.597446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.597424    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:09.599066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.599031    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e3944f-39f9-4d96-b483-9188928feea0-kube-api-access-kkq46" pod "41e3944f-39f9-4d96-b483-9188928feea0" (UID: "41e3944f-39f9-4d96-b483-9188928feea0"). InnerVolumeSpecName "kube-api-access-kkq46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:09.599186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.599070    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-xks-ca" pod "41e3944f-39f9-4d96-b483-9188928feea0" (UID: "41e3944f-39f9-4d96-b483-9188928feea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:09.698329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.698297    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e3944f-39f9-4d96-b483-9188928feea0-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:09.698329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:09.698329    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkq46\" (UniqueName: \"kubernetes.io/projected/41e3944f-39f9-4d96-b483-9188928feea0-kube-api-access-kkq46\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:10.254079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.254037    4519 generic.go:356] "Generic (PLEG): container finished" podID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerID="3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f" exitCode=0
Oct 30 20:24:10.254265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.254094    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" event={"ID":"26412140-b8b8-428b-802f-4e1a66f8fd79","Type":"ContainerDied","Data":"3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f"}
Oct 30 20:24:10.254591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.254572    4519 scope.go:122] "RemoveContainer" containerID="3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f"
Oct 30 20:24:10.256292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.256266    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t" event={"ID":"41e3944f-39f9-4d96-b483-9188928feea0","Type":"ContainerDied","Data":"975b8240b15e3e2ea5fe42e7712881558478ed75812b2f15cf7d45f662566d8d"}
Oct 30 20:24:10.256416 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.256314    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="975b8240b15e3e2ea5fe42e7712881558478ed75812b2f15cf7d45f662566d8d"
Oct 30 20:24:10.256416 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.256374    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"
Oct 30 20:24:10.256614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.256584    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:24:10.256691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.256677    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:24:10.256757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:10.256731    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:24:11.293418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:11.293370    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" event={"ID":"26412140-b8b8-428b-802f-4e1a66f8fd79","Type":"ContainerStarted","Data":"bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b"}
Oct 30 20:24:11.344386 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:11.344352    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:11.398034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:11.397990    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:24:11.398261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:11.398189    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 20:24:11.398261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:11.398231    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 20:24:12.407491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:12.398855    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 20:24:12.407491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:12.398934    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 20:24:13.398319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:13.398274    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 20:24:13.398557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:13.398357    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 20:24:13.814038 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:13.814000    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:14.434662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:14.434621    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 20:24:14.434967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:14.434933    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 20:24:15.398219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:15.398176    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 20:24:15.398709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:15.398249    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 20:24:16.412490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:16.408961    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 20:24:16.412490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:16.409060    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 20:24:17.402977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:17.402948    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:24:17.403443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:17.403429    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:24:17.404200 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:17.404180    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:24:18.729081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:18.729021    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj" podUID="de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" containerName="main" containerID="cri-o://9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d" gracePeriod=45
Oct 30 20:24:19.066028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.065980    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:24:19.209239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209204    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tmp-dir\") pod \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") "
Oct 30 20:24:19.209438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209268    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-home\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-home\") pod \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") "
Oct 30 20:24:19.209438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209309    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tls-certs\" (UniqueName: \"kubernetes.io/secret/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tls-certs\") pod \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") "
Oct 30 20:24:19.209438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209404    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-model-cache\") pod \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") "
Oct 30 20:24:19.209664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209581    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-kube-api-access-n5sqd\" (UniqueName: \"kubernetes.io/projected/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-kube-api-access-n5sqd\") pod \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") "
Oct 30 20:24:19.209664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209619    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-dshm\") pod \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\" (UID: \"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d\") "
Oct 30 20:24:19.209664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209559    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tmp-dir" pod "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" (UID: "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:19.209815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209690    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-model-cache" pod "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" (UID: "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:19.209815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.209636    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-home" pod "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" (UID: "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:19.210260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.210238    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:19.210346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.210263    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:19.210346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.210275    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:19.211739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.211708    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tls-certs" pod "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" (UID: "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:19.211862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.211779    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-kube-api-access-n5sqd" pod "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" (UID: "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d"). InnerVolumeSpecName "kube-api-access-n5sqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:19.211920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.211896    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-dshm" pod "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" (UID: "de7828ce-aeb7-4ab5-9a96-2dfbc685f37d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:19.311386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.311314    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:19.311386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.311344    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5sqd\" (UniqueName: \"kubernetes.io/projected/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-kube-api-access-n5sqd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:19.311386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.311358    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:19.404498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.404451    4519 generic.go:356] "Generic (PLEG): container finished" podID="de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" containerID="9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d" exitCode=0
Oct 30 20:24:19.404662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.404523    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"
Oct 30 20:24:19.404662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.404532    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj" event={"ID":"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d","Type":"ContainerDied","Data":"9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d"}
Oct 30 20:24:19.404662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.404580    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj" event={"ID":"de7828ce-aeb7-4ab5-9a96-2dfbc685f37d","Type":"ContainerDied","Data":"76f5c45c3b04634193ad7b5e50c31193a2f669ab9f00c600e1f175f67817c8f5"}
Oct 30 20:24:19.404662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.404598    4519 scope.go:122] "RemoveContainer" containerID="9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d"
Oct 30 20:24:19.417073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.417056    4519 scope.go:122] "RemoveContainer" containerID="9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d"
Oct 30 20:24:19.417351 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:19.417331    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d\": container with ID starting with 9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d not found: ID does not exist" containerID="9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d"
Oct 30 20:24:19.417401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.417361    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d"} err="failed to get container status \"9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d\": rpc error: code = NotFound desc = could not find container \"9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d\": container with ID starting with 9466386a629d8902abbdc77f8534d22bf5b78e8895004c5b84cf8e1ca1732e4d not found: ID does not exist"
Oct 30 20:24:19.457706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.457670    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"]
Oct 30 20:24:19.461918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.461892    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-043892/sub-test-model-043892-kserve-7895c6cb6f-dscvj"]
Oct 30 20:24:19.614439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.614361    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" path="/var/lib/kubelet/pods/de7828ce-aeb7-4ab5-9a96-2dfbc685f37d/volumes"
Oct 30 20:24:19.877811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:19.877722    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"]
Oct 30 20:24:20.978012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.977964    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"]
Oct 30 20:24:20.979634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.979600    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1fd518d9-2f66-41af-b22b-dc2d1a7be122" containerName="main"
Oct 30 20:24:20.979803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.979786    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd518d9-2f66-41af-b22b-dc2d1a7be122" containerName="main"
Oct 30 20:24:20.979957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.979946    4519 cpu_manager.go:414] "containerMap: removing container" podUID="41e3944f-39f9-4d96-b483-9188928feea0" containerName="revoke-keys"
Oct 30 20:24:20.980040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.980028    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e3944f-39f9-4d96-b483-9188928feea0" containerName="revoke-keys"
Oct 30 20:24:20.980137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.980127    4519 cpu_manager.go:414] "containerMap: removing container" podUID="de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" containerName="main"
Oct 30 20:24:20.980217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.980208    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" containerName="main"
Oct 30 20:24:20.980496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.980481    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e3944f-39f9-4d96-b483-9188928feea0" containerName="revoke-keys"
Oct 30 20:24:20.980644 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.980631    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7828ce-aeb7-4ab5-9a96-2dfbc685f37d" containerName="main"
Oct 30 20:24:20.980790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:20.980778    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd518d9-2f66-41af-b22b-dc2d1a7be122" containerName="main"
Oct 30 20:24:21.018326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.018285    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"]
Oct 30 20:24:21.018558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.018519    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.033662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.033633    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c19563-openshift-default-dockercfg-7sx4g\""
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.156448    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-data\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.156531    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-podinfo\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.156609    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-certs\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.156674    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-token\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.156986    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-envoy\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.157031    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-credential-socket\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.157178    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-socket\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.157244    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istiod-ca-cert\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.157492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.157274    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghfv\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-kube-api-access-wghfv\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.259437    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-data\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.259513    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-podinfo\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.259592    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-certs\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.259670    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-token\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.259844    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-envoy\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.259871    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-credential-socket\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.259956    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-socket\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.260010    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istiod-ca-cert\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.260539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.260039    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghfv\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-kube-api-access-wghfv\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.261165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.260845    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-data\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.270896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.270861    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-certs\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.271394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.271369    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-socket\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.271841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.271819    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-credential-socket\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.285486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.272900    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istiod-ca-cert\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.285665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.276777    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-envoy\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.285775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.280118    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-podinfo\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.331499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.331087    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-token\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.340804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.340167    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghfv\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-kube-api-access-wghfv\") pod \"e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") " pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.340983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.340920    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:21.802696 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:24:21.802666    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b38681b_3ca5_4c6b_8e5a_ff36c5569665.slice/crio-01db642a5b6713950f275508d4e44f4bcd378c82e8bc5a67626c822e39ed13da WatchSource:0}: Error finding container 01db642a5b6713950f275508d4e44f4bcd378c82e8bc5a67626c822e39ed13da: Status 404 returned error can't find the container with id 01db642a5b6713950f275508d4e44f4bcd378c82e8bc5a67626c822e39ed13da
Oct 30 20:24:21.804193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.804163    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"]
Oct 30 20:24:21.805785 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.805752    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:24:21.805853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.805829    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:24:21.805926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:21.805857    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:24:22.499762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:22.499719    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" event={"ID":"4b38681b-3ca5-4c6b-8e5a-ff36c5569665","Type":"ContainerStarted","Data":"1feaf89c42b04dbba2bd1d28a3ecadc86642385d7dc8384c3cd6258ff10c601d"}
Oct 30 20:24:22.500199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:22.499786    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" event={"ID":"4b38681b-3ca5-4c6b-8e5a-ff36c5569665","Type":"ContainerStarted","Data":"01db642a5b6713950f275508d4e44f4bcd378c82e8bc5a67626c822e39ed13da"}
Oct 30 20:24:22.558562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:22.558509    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podStartSLOduration=2.558494331 podStartE2EDuration="2.558494331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:22.554157112 +0000 UTC m=+2987.054860835" watchObservedRunningTime="2026-10-30 20:24:22.558494331 +0000 UTC m=+2987.059198055"
Oct 30 20:24:23.343151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:23.343101    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:23.352184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:23.352153    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:23.508204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:23.508157    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:23.509189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:23.509165    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:24:23.895715 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:23.895668    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:26.242683 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:26.242637    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:31.396061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.396020    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"]
Oct 30 20:24:31.415694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.415664    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"]
Oct 30 20:24:31.416009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.415991    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:31.431778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.431743    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b7eec8-dockercfg-6kmfj\""
Oct 30 20:24:31.431918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.431799    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7eec8\""
Oct 30 20:24:31.480525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.480454    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8nzc\" (UniqueName: \"kubernetes.io/projected/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-kube-api-access-f8nzc\") pod \"payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:31.480893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.480868    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:31.582641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.582608    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:31.582823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.582760    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8nzc\" (UniqueName: \"kubernetes.io/projected/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-kube-api-access-f8nzc\") pod \"payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:31.583332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.583308    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:31.603019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.602993    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8nzc\" (UniqueName: \"kubernetes.io/projected/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-kube-api-access-f8nzc\") pod \"payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:31.738991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:31.738958    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:32.053228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:32.052986    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"]
Oct 30 20:24:32.233233 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:32.233191    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:24:32.583740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:32.583702    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6" event={"ID":"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac","Type":"ContainerStarted","Data":"a697156ae662aa56638003b418b5b0c727d663ef735a34e6dadb35e4a786ccbc"}
Oct 30 20:24:32.584249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:32.583761    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6" event={"ID":"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac","Type":"ContainerStarted","Data":"c515a727c45f458525f4b08692db5adb1b557e4d68736bf0266c52d284e9924d"}
Oct 30 20:24:32.584249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:32.583796    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:32.628145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:32.628088    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6" podStartSLOduration=1.6280710919999999 podStartE2EDuration="1.628071092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:32.62439878 +0000 UTC m=+2997.125102504" watchObservedRunningTime="2026-10-30 20:24:32.628071092 +0000 UTC m=+2997.128774820"
Oct 30 20:24:33.912857 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:33.912813    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1daca35_8117_4cd3_83e6_34b2edf18c2f.slice/crio-conmon-14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:24:34.851741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:34.851697    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"]
Oct 30 20:24:34.862900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:34.862870    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-043892-hwj9t"]
Oct 30 20:24:34.922018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:34.920832    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4" podUID="e9ae9196-8b5a-4d86-840f-67e3d79a0cad" containerName="main" containerID="cri-o://339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98" gracePeriod=45
Oct 30 20:24:35.256156 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.256122    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:24:35.371340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371302    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tmp-dir\") pod \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") "
Oct 30 20:24:35.371562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371356    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-dshm\") pod \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") "
Oct 30 20:24:35.371562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371385    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-model-cache\") pod \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") "
Oct 30 20:24:35.371562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371516    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-home\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-home\") pod \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") "
Oct 30 20:24:35.371806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tls-certs\") pod \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") "
Oct 30 20:24:35.371806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371667    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-kube-api-access-d8nxn\" (UniqueName: \"kubernetes.io/projected/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-kube-api-access-d8nxn\") pod \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\" (UID: \"e9ae9196-8b5a-4d86-840f-67e3d79a0cad\") "
Oct 30 20:24:35.371806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371702    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tmp-dir" pod "e9ae9196-8b5a-4d86-840f-67e3d79a0cad" (UID: "e9ae9196-8b5a-4d86-840f-67e3d79a0cad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:35.371806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371743    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-model-cache" pod "e9ae9196-8b5a-4d86-840f-67e3d79a0cad" (UID: "e9ae9196-8b5a-4d86-840f-67e3d79a0cad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:35.372046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.371924    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-home" pod "e9ae9196-8b5a-4d86-840f-67e3d79a0cad" (UID: "e9ae9196-8b5a-4d86-840f-67e3d79a0cad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:35.372264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.372246    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:35.372346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.372270    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:35.372346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.372283    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:35.373808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.373783    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-dshm" pod "e9ae9196-8b5a-4d86-840f-67e3d79a0cad" (UID: "e9ae9196-8b5a-4d86-840f-67e3d79a0cad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:35.374144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.374124    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-kube-api-access-d8nxn" pod "e9ae9196-8b5a-4d86-840f-67e3d79a0cad" (UID: "e9ae9196-8b5a-4d86-840f-67e3d79a0cad"). InnerVolumeSpecName "kube-api-access-d8nxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:35.374305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.374280    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tls-certs" pod "e9ae9196-8b5a-4d86-840f-67e3d79a0cad" (UID: "e9ae9196-8b5a-4d86-840f-67e3d79a0cad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:35.473310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.473274    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:35.473310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.473308    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8nxn\" (UniqueName: \"kubernetes.io/projected/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-kube-api-access-d8nxn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:35.473528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.473321    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ae9196-8b5a-4d86-840f-67e3d79a0cad-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:35.614136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.614103    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e3944f-39f9-4d96-b483-9188928feea0" path="/var/lib/kubelet/pods/41e3944f-39f9-4d96-b483-9188928feea0/volumes"
Oct 30 20:24:35.616748 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.616723    4519 generic.go:356] "Generic (PLEG): container finished" podID="e9ae9196-8b5a-4d86-840f-67e3d79a0cad" containerID="339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98" exitCode=0
Oct 30 20:24:35.616895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.616758    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"
Oct 30 20:24:35.616895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.616801    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4" event={"ID":"e9ae9196-8b5a-4d86-840f-67e3d79a0cad","Type":"ContainerDied","Data":"339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98"}
Oct 30 20:24:35.616895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.616837    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4" event={"ID":"e9ae9196-8b5a-4d86-840f-67e3d79a0cad","Type":"ContainerDied","Data":"2306733f7fc329614bfe68850d953e805f1c18bb602df75513374381dbd23438"}
Oct 30 20:24:35.616895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.616859    4519 scope.go:122] "RemoveContainer" containerID="339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98"
Oct 30 20:24:35.634769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.634708    4519 scope.go:122] "RemoveContainer" containerID="339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98"
Oct 30 20:24:35.635102 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:35.635080    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98\": container with ID starting with 339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98 not found: ID does not exist" containerID="339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98"
Oct 30 20:24:35.635174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.635109    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98"} err="failed to get container status \"339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98\": rpc error: code = NotFound desc = could not find container \"339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98\": container with ID starting with 339ab5660b31db75c64b348166d38d107c1e1f38bc89acbb56596a70433f1d98 not found: ID does not exist"
Oct 30 20:24:35.676878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.676843    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"]
Oct 30 20:24:35.686509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:35.686461    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4"]
Oct 30 20:24:36.208262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:36.208216    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-27a4d3/sub-test-model-27a4d3-kserve-6fb9bf459f-d5np4" podUID="e9ae9196-8b5a-4d86-840f-67e3d79a0cad" containerName="main" probeResult="failure" output="Get \"https://10.133.1.16:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:24:37.615571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:37.615535    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ae9196-8b5a-4d86-840f-67e3d79a0cad" path="/var/lib/kubelet/pods/e9ae9196-8b5a-4d86-840f-67e3d79a0cad/volumes"
Oct 30 20:24:42.404137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:42.404103    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"]
Oct 30 20:24:42.404619 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:42.404336    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" containerID="cri-o://7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b" gracePeriod=30
Oct 30 20:24:42.996950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:42.996916    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"]
Oct 30 20:24:42.997602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:42.997578    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e9ae9196-8b5a-4d86-840f-67e3d79a0cad" containerName="main"
Oct 30 20:24:42.997602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:42.997596    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ae9196-8b5a-4d86-840f-67e3d79a0cad" containerName="main"
Oct 30 20:24:42.997812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:42.997703    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ae9196-8b5a-4d86-840f-67e3d79a0cad" containerName="main"
Oct 30 20:24:43.000925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.000904    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.015515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.015456    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:43.015701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.015683    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:24:43.021845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.021817    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"]
Oct 30 20:24:43.025039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.025006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv52l\" (UniqueName: \"kubernetes.io/projected/b5f94067-7553-4df9-81f9-b684211e30c3-kube-api-access-tv52l\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.025226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.025207    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.025299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.025252    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.126840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.126804    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv52l\" (UniqueName: \"kubernetes.io/projected/b5f94067-7553-4df9-81f9-b684211e30c3-kube-api-access-tv52l\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.127015 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.126950    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.127015 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.126984    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.127670 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.127644    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.129458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.129432    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.146327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.146299    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv52l\" (UniqueName: \"kubernetes.io/projected/b5f94067-7553-4df9-81f9-b684211e30c3-kube-api-access-tv52l\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.317451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.317370    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:43.597141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.597059    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:24:43.668854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.668820    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"]
Oct 30 20:24:43.728620 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:43.728590    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx" event={"ID":"b5f94067-7553-4df9-81f9-b684211e30c3","Type":"ContainerStarted","Data":"6d8c41b108d6801cb0bf89eece917ae76054ef6e45521c5969217e8d0db2620b"}
Oct 30 20:24:44.030321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:44.030280    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"]
Oct 30 20:24:44.030531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:44.030506    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy" containerID="cri-o://0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144" gracePeriod=30
Oct 30 20:24:44.756076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:44.756042    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx" event={"ID":"b5f94067-7553-4df9-81f9-b684211e30c3","Type":"ContainerStarted","Data":"1d8190abe702efdd744b887ece4b86892cdd169c46601e9020746f4c97a34efa"}
Oct 30 20:24:44.847459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:44.847417    4519 scope.go:122] "RemoveContainer" containerID="f2f4ac3f20c967991272834d94dd4b02126005acdf84cdf7356c89409ddd651e"
Oct 30 20:24:44.859759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:44.859670    4519 scope.go:122] "RemoveContainer" containerID="a246f4ba9cde4925b2e1f7efaec03d978b4d982ccfd2c3faa1980a010edd4831"
Oct 30 20:24:44.870376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:44.870185    4519 scope.go:122] "RemoveContainer" containerID="915ec22dd49106bef7bb322a678a510b1b0a929f206cf1d157e5e38e86afd4fc"
Oct 30 20:24:45.353904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.353862    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:24:45.354107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.353928    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:24:45.778984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.778940    4519 generic.go:356] "Generic (PLEG): container finished" podID="b5f94067-7553-4df9-81f9-b684211e30c3" containerID="1d8190abe702efdd744b887ece4b86892cdd169c46601e9020746f4c97a34efa" exitCode=0
Oct 30 20:24:45.779526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.779025    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx" event={"ID":"b5f94067-7553-4df9-81f9-b684211e30c3","Type":"ContainerDied","Data":"1d8190abe702efdd744b887ece4b86892cdd169c46601e9020746f4c97a34efa"}
Oct 30 20:24:45.902884 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.902846    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"]
Oct 30 20:24:45.909591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.909565    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:45.938521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.927845    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"]
Oct 30 20:24:45.978523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.978484    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mffq\" (UniqueName: \"kubernetes.io/projected/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-kube-api-access-8mffq\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:45.978710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.978571    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:45.978710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.978652    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:45.980129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:45.980108    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:46.080507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.080370    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-xks-ca\") pod \"b5f94067-7553-4df9-81f9-b684211e30c3\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") "
Oct 30 20:24:46.080681 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.080509    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f94067-7553-4df9-81f9-b684211e30c3-kube-api-access-tv52l\" (UniqueName: \"kubernetes.io/projected/b5f94067-7553-4df9-81f9-b684211e30c3-kube-api-access-tv52l\") pod \"b5f94067-7553-4df9-81f9-b684211e30c3\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") "
Oct 30 20:24:46.080681 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.080563    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-openshift-ca\") pod \"b5f94067-7553-4df9-81f9-b684211e30c3\" (UID: \"b5f94067-7553-4df9-81f9-b684211e30c3\") "
Oct 30 20:24:46.080945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.080920    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mffq\" (UniqueName: \"kubernetes.io/projected/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-kube-api-access-8mffq\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:46.081081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.080970    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:46.081081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.080996    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-openshift-ca" pod "b5f94067-7553-4df9-81f9-b684211e30c3" (UID: "b5f94067-7553-4df9-81f9-b684211e30c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:46.081204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.081083    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:46.081378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.081334    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:46.081850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.081823    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:46.083436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.082901    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-xks-ca" pod "b5f94067-7553-4df9-81f9-b684211e30c3" (UID: "b5f94067-7553-4df9-81f9-b684211e30c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:46.083436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.082935    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f94067-7553-4df9-81f9-b684211e30c3-kube-api-access-tv52l" pod "b5f94067-7553-4df9-81f9-b684211e30c3" (UID: "b5f94067-7553-4df9-81f9-b684211e30c3"). InnerVolumeSpecName "kube-api-access-tv52l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:46.084071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.084046    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:46.100914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.100889    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mffq\" (UniqueName: \"kubernetes.io/projected/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-kube-api-access-8mffq\") pod \"maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:46.183533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.183485    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f94067-7553-4df9-81f9-b684211e30c3-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:46.183533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.183526    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv52l\" (UniqueName: \"kubernetes.io/projected/b5f94067-7553-4df9-81f9-b684211e30c3-kube-api-access-tv52l\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:46.229647 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.229616    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:46.562074 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:24:46.562036    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63494403_7ac3_4c5f_b66c_ae3e9f6110e9.slice/crio-d718afd423376abce5ac40ba26c739a6e78512c56981a83fd46b03246a9ddaf0 WatchSource:0}: Error finding container d718afd423376abce5ac40ba26c739a6e78512c56981a83fd46b03246a9ddaf0: Status 404 returned error can't find the container with id d718afd423376abce5ac40ba26c739a6e78512c56981a83fd46b03246a9ddaf0
Oct 30 20:24:46.565401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.565367    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"]
Oct 30 20:24:46.789937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.789903    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"
Oct 30 20:24:46.790455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.789910    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx" event={"ID":"b5f94067-7553-4df9-81f9-b684211e30c3","Type":"ContainerDied","Data":"6d8c41b108d6801cb0bf89eece917ae76054ef6e45521c5969217e8d0db2620b"}
Oct 30 20:24:46.790455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.790037    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8c41b108d6801cb0bf89eece917ae76054ef6e45521c5969217e8d0db2620b"
Oct 30 20:24:46.791773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:46.791734    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw" event={"ID":"63494403-7ac3-4c5f-b66c-ae3e9f6110e9","Type":"ContainerStarted","Data":"d718afd423376abce5ac40ba26c739a6e78512c56981a83fd46b03246a9ddaf0"}
Oct 30 20:24:47.707723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.707690    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:24:47.806169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.806127    4519 generic.go:356] "Generic (PLEG): container finished" podID="63494403-7ac3-4c5f-b66c-ae3e9f6110e9" containerID="ff663d81dce5ddf829025d656168fdcef41f36f06c2f85397e64176b9efc3800" exitCode=0
Oct 30 20:24:47.806723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.806210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw" event={"ID":"63494403-7ac3-4c5f-b66c-ae3e9f6110e9","Type":"ContainerDied","Data":"ff663d81dce5ddf829025d656168fdcef41f36f06c2f85397e64176b9efc3800"}
Oct 30 20:24:47.808244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.808218    4519 generic.go:356] "Generic (PLEG): container finished" podID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerID="7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b" exitCode=0
Oct 30 20:24:47.808393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.808256    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"
Oct 30 20:24:47.808393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.808289    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" event={"ID":"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f","Type":"ContainerDied","Data":"7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b"}
Oct 30 20:24:47.808393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.808357    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc" event={"ID":"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f","Type":"ContainerDied","Data":"56b83b10f1177a6752bd3e950c5e3ea20d4d51f8ef84411e5a85c7ce35d402ca"}
Oct 30 20:24:47.808539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.808392    4519 scope.go:122] "RemoveContainer" containerID="7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b"
Oct 30 20:24:47.813205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813177    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-kube-api-access-km85r\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813221    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-credential-socket\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813295    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-certs\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813332    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-envoy\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813417    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-socket\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813570    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-token\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813684    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-credential-socket" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:47.813711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813707    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-data\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813797    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istiod-ca-cert\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.813941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813917    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-data" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:47.813941 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.813932    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-podinfo\") pod \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\" (UID: \"a1159ab9-aa23-471f-88ff-ddf2bfe25b1f\") "
Oct 30 20:24:47.814149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.814127    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-certs" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:47.814594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.814460    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istiod-ca-cert" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:47.814717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.814690    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.815992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.814717    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.815992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.814731    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.815992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.814744    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.815992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.815071    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-socket" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:47.817719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.817595    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-envoy" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:47.818026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.818000    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-podinfo" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:24:47.818272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.818252    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-token" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:47.819295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.819273    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-kube-api-access-km85r" pod "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" (UID: "a1159ab9-aa23-471f-88ff-ddf2bfe25b1f"). InnerVolumeSpecName "kube-api-access-km85r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:47.823036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.823019    4519 scope.go:122] "RemoveContainer" containerID="7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b"
Oct 30 20:24:47.823344 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:47.823322    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b\": container with ID starting with 7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b not found: ID does not exist" containerID="7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b"
Oct 30 20:24:47.823402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.823352    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b"} err="failed to get container status \"7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b\": rpc error: code = NotFound desc = could not find container \"7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b\": container with ID starting with 7691383499c15cd0c8dcb5672192a5352a39a0a47bb5931460a577d048ea619b not found: ID does not exist"
Oct 30 20:24:47.916185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.916152    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.916185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.916184    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-kube-api-access-km85r\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.916379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.916194    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.916379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.916205    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:47.916379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:47.916215    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:48.161298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:48.161258    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"]
Oct 30 20:24:48.170080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:48.170045    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-7rmdc"]
Oct 30 20:24:49.014001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.013969    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:49.138803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.138721    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-openshift-ca\") pod \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") "
Oct 30 20:24:49.139001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.138987    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-kube-api-access-8mffq\" (UniqueName: \"kubernetes.io/projected/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-kube-api-access-8mffq\") pod \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") "
Oct 30 20:24:49.139066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.139050    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-xks-ca\") pod \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\" (UID: \"63494403-7ac3-4c5f-b66c-ae3e9f6110e9\") "
Oct 30 20:24:49.139155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.139127    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-openshift-ca" pod "63494403-7ac3-4c5f-b66c-ae3e9f6110e9" (UID: "63494403-7ac3-4c5f-b66c-ae3e9f6110e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:49.139692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.139669    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.141933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.141905    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-xks-ca" pod "63494403-7ac3-4c5f-b66c-ae3e9f6110e9" (UID: "63494403-7ac3-4c5f-b66c-ae3e9f6110e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:49.145858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.145832    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-kube-api-access-8mffq" pod "63494403-7ac3-4c5f-b66c-ae3e9f6110e9" (UID: "63494403-7ac3-4c5f-b66c-ae3e9f6110e9"). InnerVolumeSpecName "kube-api-access-8mffq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:49.241064 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.241020    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mffq\" (UniqueName: \"kubernetes.io/projected/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-kube-api-access-8mffq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.241064 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.241060    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63494403-7ac3-4c5f-b66c-ae3e9f6110e9-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.310881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.310856    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 20:24:49.342077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342048    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9045b774-bcc5-4441-bf11-8c3dba36f363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9045b774-bcc5-4441-bf11-8c3dba36f363-istiod-ca-cert\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342233 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-kube-api-access-h5wgf\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-kube-api-access-h5wgf\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342233 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342169    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-podinfo\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342233 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342210    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-socket\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342322    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-data\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342375    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-credential-socket\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342483 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342402    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-token\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-token\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342483 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342434    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-certs\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342483 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342443    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9045b774-bcc5-4441-bf11-8c3dba36f363-istiod-ca-cert" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:49.342647 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342529    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-envoy\") pod \"9045b774-bcc5-4441-bf11-8c3dba36f363\" (UID: \"9045b774-bcc5-4441-bf11-8c3dba36f363\") "
Oct 30 20:24:49.342750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342579    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-socket" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:49.342833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342652    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-data" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:49.342899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342841    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-certs" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:49.342943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.342915    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-credential-socket" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:49.343496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.343447    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9045b774-bcc5-4441-bf11-8c3dba36f363-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.343632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.343513    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.343632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.343528    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.343632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.343540    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.343632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.343554    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.344788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.344768    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-kube-api-access-h5wgf" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "kube-api-access-h5wgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:49.344867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.344827    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-podinfo" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:24:49.345025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.344996    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-token" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:49.345130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.345072    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-envoy" pod "9045b774-bcc5-4441-bf11-8c3dba36f363" (UID: "9045b774-bcc5-4441-bf11-8c3dba36f363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:49.445138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.445108    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.445138 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.445136    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.445339 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.445146    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5wgf\" (UniqueName: \"kubernetes.io/projected/9045b774-bcc5-4441-bf11-8c3dba36f363-kube-api-access-h5wgf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.445339 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.445157    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9045b774-bcc5-4441-bf11-8c3dba36f363-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:49.614344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.614306    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" path="/var/lib/kubelet/pods/a1159ab9-aa23-471f-88ff-ddf2bfe25b1f/volumes"
Oct 30 20:24:49.837652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.837568    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"
Oct 30 20:24:49.837652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.837605    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw" event={"ID":"63494403-7ac3-4c5f-b66c-ae3e9f6110e9","Type":"ContainerDied","Data":"d718afd423376abce5ac40ba26c739a6e78512c56981a83fd46b03246a9ddaf0"}
Oct 30 20:24:49.837863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.837655    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d718afd423376abce5ac40ba26c739a6e78512c56981a83fd46b03246a9ddaf0"
Oct 30 20:24:49.839245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.839214    4519 generic.go:356] "Generic (PLEG): container finished" podID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerID="0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144" exitCode=0
Oct 30 20:24:49.839368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.839248    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" event={"ID":"9045b774-bcc5-4441-bf11-8c3dba36f363","Type":"ContainerDied","Data":"0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144"}
Oct 30 20:24:49.839368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.839290    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"
Oct 30 20:24:49.839368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.839313    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx" event={"ID":"9045b774-bcc5-4441-bf11-8c3dba36f363","Type":"ContainerDied","Data":"b30ab067d086f068b2f9b72c537f9fdbf10b78ae071907c4553102f490dbfae2"}
Oct 30 20:24:49.839368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.839342    4519 scope.go:122] "RemoveContainer" containerID="0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144"
Oct 30 20:24:49.850129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.850095    4519 scope.go:122] "RemoveContainer" containerID="0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144"
Oct 30 20:24:49.850447 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:49.850423    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144\": container with ID starting with 0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144 not found: ID does not exist" containerID="0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144"
Oct 30 20:24:49.850552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.850458    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144"} err="failed to get container status \"0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144\": rpc error: code = NotFound desc = could not find container \"0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144\": container with ID starting with 0ad9a014a31eaa9f5fba4cb80571c23f7110ed1a04a114bae7eaca0bf6f17144 not found: ID does not exist"
Oct 30 20:24:49.910719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.910691    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"]
Oct 30 20:24:49.921957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:49.921926    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c2xqx"]
Oct 30 20:24:51.613337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:51.613301    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" path="/var/lib/kubelet/pods/9045b774-bcc5-4441-bf11-8c3dba36f363/volumes"
Oct 30 20:24:57.216804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.216749    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"]
Oct 30 20:24:57.217427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.217370    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt" podUID="98453c2c-4cb9-482f-9638-55553dfc6e8e" containerName="payload-processing" containerID="cri-o://cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6" gracePeriod=30
Oct 30 20:24:57.483080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.482326    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"]
Oct 30 20:24:57.493667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.489633    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-159b3a-kbqfx"]
Oct 30 20:24:57.600289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.600258    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:24:57.694491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.694231    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98453c2c-4cb9-482f-9638-55553dfc6e8e-kube-api-access-gd6hj\" (UniqueName: \"kubernetes.io/projected/98453c2c-4cb9-482f-9638-55553dfc6e8e-kube-api-access-gd6hj\") pod \"98453c2c-4cb9-482f-9638-55553dfc6e8e\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") "
Oct 30 20:24:57.694491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.694316    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98453c2c-4cb9-482f-9638-55553dfc6e8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98453c2c-4cb9-482f-9638-55553dfc6e8e-plugins-config-volume\") pod \"98453c2c-4cb9-482f-9638-55553dfc6e8e\" (UID: \"98453c2c-4cb9-482f-9638-55553dfc6e8e\") "
Oct 30 20:24:57.711485 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.703716    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98453c2c-4cb9-482f-9638-55553dfc6e8e-plugins-config-volume" pod "98453c2c-4cb9-482f-9638-55553dfc6e8e" (UID: "98453c2c-4cb9-482f-9638-55553dfc6e8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:57.717490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.716490    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f94067-7553-4df9-81f9-b684211e30c3" path="/var/lib/kubelet/pods/b5f94067-7553-4df9-81f9-b684211e30c3/volumes"
Oct 30 20:24:57.736584 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.730417    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98453c2c-4cb9-482f-9638-55553dfc6e8e-kube-api-access-gd6hj" pod "98453c2c-4cb9-482f-9638-55553dfc6e8e" (UID: "98453c2c-4cb9-482f-9638-55553dfc6e8e"). InnerVolumeSpecName "kube-api-access-gd6hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:57.807095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.807051    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd6hj\" (UniqueName: \"kubernetes.io/projected/98453c2c-4cb9-482f-9638-55553dfc6e8e-kube-api-access-gd6hj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:57.807095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:57.807094    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98453c2c-4cb9-482f-9638-55553dfc6e8e-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:24:58.014851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.014745    4519 generic.go:356] "Generic (PLEG): container finished" podID="98453c2c-4cb9-482f-9638-55553dfc6e8e" containerID="cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6" exitCode=0
Oct 30 20:24:58.015339 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.015311    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt" event={"ID":"98453c2c-4cb9-482f-9638-55553dfc6e8e","Type":"ContainerDied","Data":"cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6"}
Oct 30 20:24:58.015586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.015552    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt" event={"ID":"98453c2c-4cb9-482f-9638-55553dfc6e8e","Type":"ContainerDied","Data":"f62438518052274216e3f7c1e31aeb6213ef86e544d7b71609af2b46465fa871"}
Oct 30 20:24:58.015762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.015735    4519 scope.go:122] "RemoveContainer" containerID="cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6"
Oct 30 20:24:58.015951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.015931    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"
Oct 30 20:24:58.040277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.040251    4519 scope.go:122] "RemoveContainer" containerID="cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6"
Oct 30 20:24:58.040654 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:24:58.040624    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6\": container with ID starting with cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6 not found: ID does not exist" containerID="cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6"
Oct 30 20:24:58.040774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.040662    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6"} err="failed to get container status \"cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6\": rpc error: code = NotFound desc = could not find container \"cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6\": container with ID starting with cda0b1cfdb7651170ae86459fc9906e0ff7be4a67d47ebef4df7b64e681590c6 not found: ID does not exist"
Oct 30 20:24:58.103491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.099558    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"]
Oct 30 20:24:58.107080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.107025    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-584b7b5fb6-rkpkt"]
Oct 30 20:24:58.391376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.391268    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"]
Oct 30 20:24:58.398183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.398143    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6175d5-b5ktw"]
Oct 30 20:24:58.939171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939132    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"]
Oct 30 20:24:58.939779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939762    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy"
Oct 30 20:24:58.939875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939780    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy"
Oct 30 20:24:58.939875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939820    4519 cpu_manager.go:414] "containerMap: removing container" podUID="98453c2c-4cb9-482f-9638-55553dfc6e8e" containerName="payload-processing"
Oct 30 20:24:58.939875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939825    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98453c2c-4cb9-482f-9638-55553dfc6e8e" containerName="payload-processing"
Oct 30 20:24:58.939875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939856    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy"
Oct 30 20:24:58.939875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939862    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy"
Oct 30 20:24:58.939875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939873    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b5f94067-7553-4df9-81f9-b684211e30c3" containerName="revoke-keys"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939878    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f94067-7553-4df9-81f9-b684211e30c3" containerName="revoke-keys"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939888    4519 cpu_manager.go:414] "containerMap: removing container" podUID="63494403-7ac3-4c5f-b66c-ae3e9f6110e9" containerName="revoke-keys"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939892    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63494403-7ac3-4c5f-b66c-ae3e9f6110e9" containerName="revoke-keys"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939988    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9045b774-bcc5-4441-bf11-8c3dba36f363" containerName="istio-proxy"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.939997    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="98453c2c-4cb9-482f-9638-55553dfc6e8e" containerName="payload-processing"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.940008    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1159ab9-aa23-471f-88ff-ddf2bfe25b1f" containerName="istio-proxy"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.940015    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f94067-7553-4df9-81f9-b684211e30c3" containerName="revoke-keys"
Oct 30 20:24:58.940073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.940021    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="63494403-7ac3-4c5f-b66c-ae3e9f6110e9" containerName="revoke-keys"
Oct 30 20:24:58.945069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.945040    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:58.970509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.970454    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"]
Oct 30 20:24:58.976561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.976523    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:58.976825 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:58.976801    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:24:59.034263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.034229    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.034501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.034450    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.034570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.034535    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz86v\" (UniqueName: \"kubernetes.io/projected/075d9333-ef02-4c2d-8d9b-03551f02a8e7-kube-api-access-hz86v\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.136199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.136152    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.136417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.136398    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.136521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.136452    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz86v\" (UniqueName: \"kubernetes.io/projected/075d9333-ef02-4c2d-8d9b-03551f02a8e7-kube-api-access-hz86v\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.137116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.137094    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.139014 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.138991    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.165933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.165905    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz86v\" (UniqueName: \"kubernetes.io/projected/075d9333-ef02-4c2d-8d9b-03551f02a8e7-kube-api-access-hz86v\") pod \"maas-api-revoke-sub-1b336b3b-wt2tt\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.260057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.259981    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:24:59.571824 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:24:59.571786    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075d9333_ef02_4c2d_8d9b_03551f02a8e7.slice/crio-d30b3a8b60d9252b5ae40e8a72f975212d6e1645908e2e53f951546330b68153 WatchSource:0}: Error finding container d30b3a8b60d9252b5ae40e8a72f975212d6e1645908e2e53f951546330b68153: Status 404 returned error can't find the container with id d30b3a8b60d9252b5ae40e8a72f975212d6e1645908e2e53f951546330b68153
Oct 30 20:24:59.572288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.572002    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"]
Oct 30 20:24:59.614773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.614739    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63494403-7ac3-4c5f-b66c-ae3e9f6110e9" path="/var/lib/kubelet/pods/63494403-7ac3-4c5f-b66c-ae3e9f6110e9/volumes"
Oct 30 20:24:59.615257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:24:59.615239    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98453c2c-4cb9-482f-9638-55553dfc6e8e" path="/var/lib/kubelet/pods/98453c2c-4cb9-482f-9638-55553dfc6e8e/volumes"
Oct 30 20:25:00.037283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.037236    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt" event={"ID":"075d9333-ef02-4c2d-8d9b-03551f02a8e7","Type":"ContainerStarted","Data":"51bf4039654de26a597de7067c7428a688504fea47bd5d520919fd6536e4cad3"}
Oct 30 20:25:00.037533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.037302    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt" event={"ID":"075d9333-ef02-4c2d-8d9b-03551f02a8e7","Type":"ContainerStarted","Data":"d30b3a8b60d9252b5ae40e8a72f975212d6e1645908e2e53f951546330b68153"}
Oct 30 20:25:00.080096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.080041    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt" podStartSLOduration=2.080024842 podStartE2EDuration="2.080024842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:00.077830578 +0000 UTC m=+3024.578534301" watchObservedRunningTime="2026-10-30 20:25:00.080024842 +0000 UTC m=+3024.580728566"
Oct 30 20:25:00.476687 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.476645    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"]
Oct 30 20:25:00.481286 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.481260    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.512632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.503530    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"]
Oct 30 20:25:00.558039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.557997    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.558219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.558080    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6m9\" (UniqueName: \"kubernetes.io/projected/13d6fdd3-558e-4629-a380-95ff34ff75ff-kube-api-access-4z6m9\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.558219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.558179    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.660456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.660405    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.661004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.660533    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6m9\" (UniqueName: \"kubernetes.io/projected/13d6fdd3-558e-4629-a380-95ff34ff75ff-kube-api-access-4z6m9\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.661004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.660641    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.661259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.661218    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.663744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.663721    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.683428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.683390    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6m9\" (UniqueName: \"kubernetes.io/projected/13d6fdd3-558e-4629-a380-95ff34ff75ff-kube-api-access-4z6m9\") pod \"maas-api-revoke-sub-8cca7f94-n55nz\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:00.796925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:00.796834    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:01.047055 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:01.046954    4519 generic.go:356] "Generic (PLEG): container finished" podID="075d9333-ef02-4c2d-8d9b-03551f02a8e7" containerID="51bf4039654de26a597de7067c7428a688504fea47bd5d520919fd6536e4cad3" exitCode=0
Oct 30 20:25:01.047055 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:01.047042    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt" event={"ID":"075d9333-ef02-4c2d-8d9b-03551f02a8e7","Type":"ContainerDied","Data":"51bf4039654de26a597de7067c7428a688504fea47bd5d520919fd6536e4cad3"}
Oct 30 20:25:01.158781 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:01.158726    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d6fdd3_558e_4629_a380_95ff34ff75ff.slice/crio-f23dec5210a9409b368e7287d6897f9f59ed8e28c3a39778c89b0fcd9de957ad WatchSource:0}: Error finding container f23dec5210a9409b368e7287d6897f9f59ed8e28c3a39778c89b0fcd9de957ad: Status 404 returned error can't find the container with id f23dec5210a9409b368e7287d6897f9f59ed8e28c3a39778c89b0fcd9de957ad
Oct 30 20:25:01.165335 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:01.165242    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"]
Oct 30 20:25:02.060639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.060603    4519 generic.go:356] "Generic (PLEG): container finished" podID="13d6fdd3-558e-4629-a380-95ff34ff75ff" containerID="344d7bde428a368ad656d183eec0fd4345db715bd4df18a2351eaf9a5cd0e6b4" exitCode=0
Oct 30 20:25:02.061115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.060695    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz" event={"ID":"13d6fdd3-558e-4629-a380-95ff34ff75ff","Type":"ContainerDied","Data":"344d7bde428a368ad656d183eec0fd4345db715bd4df18a2351eaf9a5cd0e6b4"}
Oct 30 20:25:02.061115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.060762    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz" event={"ID":"13d6fdd3-558e-4629-a380-95ff34ff75ff","Type":"ContainerStarted","Data":"f23dec5210a9409b368e7287d6897f9f59ed8e28c3a39778c89b0fcd9de957ad"}
Oct 30 20:25:02.225655 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:02.225561    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:25:02.235094 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.235065    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:25:02.285561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.285524    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075d9333-ef02-4c2d-8d9b-03551f02a8e7-kube-api-access-hz86v\" (UniqueName: \"kubernetes.io/projected/075d9333-ef02-4c2d-8d9b-03551f02a8e7-kube-api-access-hz86v\") pod \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") "
Oct 30 20:25:02.285760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.285657    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-openshift-ca\") pod \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") "
Oct 30 20:25:02.285760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.285694    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-xks-ca\") pod \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\" (UID: \"075d9333-ef02-4c2d-8d9b-03551f02a8e7\") "
Oct 30 20:25:02.287957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.287874    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-openshift-ca" pod "075d9333-ef02-4c2d-8d9b-03551f02a8e7" (UID: "075d9333-ef02-4c2d-8d9b-03551f02a8e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:02.291586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.289963    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-xks-ca" pod "075d9333-ef02-4c2d-8d9b-03551f02a8e7" (UID: "075d9333-ef02-4c2d-8d9b-03551f02a8e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:02.291586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.290444    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075d9333-ef02-4c2d-8d9b-03551f02a8e7-kube-api-access-hz86v" pod "075d9333-ef02-4c2d-8d9b-03551f02a8e7" (UID: "075d9333-ef02-4c2d-8d9b-03551f02a8e7"). InnerVolumeSpecName "kube-api-access-hz86v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:02.387510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.387461    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:02.387510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.387505    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d9333-ef02-4c2d-8d9b-03551f02a8e7-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:02.387510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:02.387517    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz86v\" (UniqueName: \"kubernetes.io/projected/075d9333-ef02-4c2d-8d9b-03551f02a8e7-kube-api-access-hz86v\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:03.071041 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.071004    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"
Oct 30 20:25:03.071041 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.071026    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt" event={"ID":"075d9333-ef02-4c2d-8d9b-03551f02a8e7","Type":"ContainerDied","Data":"d30b3a8b60d9252b5ae40e8a72f975212d6e1645908e2e53f951546330b68153"}
Oct 30 20:25:03.071685 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.071099    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30b3a8b60d9252b5ae40e8a72f975212d6e1645908e2e53f951546330b68153"
Oct 30 20:25:03.243866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.243833    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:03.307643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.307598    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d6fdd3-558e-4629-a380-95ff34ff75ff-kube-api-access-4z6m9\" (UniqueName: \"kubernetes.io/projected/13d6fdd3-558e-4629-a380-95ff34ff75ff-kube-api-access-4z6m9\") pod \"13d6fdd3-558e-4629-a380-95ff34ff75ff\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") "
Oct 30 20:25:03.307841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.307806    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-xks-ca\") pod \"13d6fdd3-558e-4629-a380-95ff34ff75ff\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") "
Oct 30 20:25:03.307987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.307967    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-openshift-ca\") pod \"13d6fdd3-558e-4629-a380-95ff34ff75ff\" (UID: \"13d6fdd3-558e-4629-a380-95ff34ff75ff\") "
Oct 30 20:25:03.308488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.308340    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-openshift-ca" pod "13d6fdd3-558e-4629-a380-95ff34ff75ff" (UID: "13d6fdd3-558e-4629-a380-95ff34ff75ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:03.308690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.308665    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:03.309944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.309914    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d6fdd3-558e-4629-a380-95ff34ff75ff-kube-api-access-4z6m9" pod "13d6fdd3-558e-4629-a380-95ff34ff75ff" (UID: "13d6fdd3-558e-4629-a380-95ff34ff75ff"). InnerVolumeSpecName "kube-api-access-4z6m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:03.309944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.309917    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-xks-ca" pod "13d6fdd3-558e-4629-a380-95ff34ff75ff" (UID: "13d6fdd3-558e-4629-a380-95ff34ff75ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:03.410461 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.410413    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z6m9\" (UniqueName: \"kubernetes.io/projected/13d6fdd3-558e-4629-a380-95ff34ff75ff-kube-api-access-4z6m9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:03.410461 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:03.410460    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d6fdd3-558e-4629-a380-95ff34ff75ff-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:04.085572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:04.085542    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"
Oct 30 20:25:04.086130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:04.085545    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz" event={"ID":"13d6fdd3-558e-4629-a380-95ff34ff75ff","Type":"ContainerDied","Data":"f23dec5210a9409b368e7287d6897f9f59ed8e28c3a39778c89b0fcd9de957ad"}
Oct 30 20:25:04.086287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:04.086159    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f23dec5210a9409b368e7287d6897f9f59ed8e28c3a39778c89b0fcd9de957ad"
Oct 30 20:25:05.644406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:05.644364    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"]
Oct 30 20:25:05.644890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:05.644669    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy" containerID="cri-o://64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b" gracePeriod=30
Oct 30 20:25:06.463594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.463553    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"]
Oct 30 20:25:06.464225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.464205    4519 cpu_manager.go:414] "containerMap: removing container" podUID="075d9333-ef02-4c2d-8d9b-03551f02a8e7" containerName="revoke-keys"
Oct 30 20:25:06.464318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.464225    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075d9333-ef02-4c2d-8d9b-03551f02a8e7" containerName="revoke-keys"
Oct 30 20:25:06.464318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.464253    4519 cpu_manager.go:414] "containerMap: removing container" podUID="13d6fdd3-558e-4629-a380-95ff34ff75ff" containerName="revoke-keys"
Oct 30 20:25:06.464318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.464258    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d6fdd3-558e-4629-a380-95ff34ff75ff" containerName="revoke-keys"
Oct 30 20:25:06.464499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.464347    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="075d9333-ef02-4c2d-8d9b-03551f02a8e7" containerName="revoke-keys"
Oct 30 20:25:06.464499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.464361    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d6fdd3-558e-4629-a380-95ff34ff75ff" containerName="revoke-keys"
Oct 30 20:25:06.467955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.467929    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.497394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.497356    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"]
Oct 30 20:25:06.498122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.497912    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:25:06.498122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.497951    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:25:06.502040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.502012    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krh5\" (UniqueName: \"kubernetes.io/projected/56483207-d997-447f-831e-5906fe8164c5-kube-api-access-6krh5\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.502336 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.502315    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56483207-d997-447f-831e-5906fe8164c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.502438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.502420    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56483207-d997-447f-831e-5906fe8164c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.626490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.606344    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krh5\" (UniqueName: \"kubernetes.io/projected/56483207-d997-447f-831e-5906fe8164c5-kube-api-access-6krh5\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.626490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.606641    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56483207-d997-447f-831e-5906fe8164c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.626490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.606713    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56483207-d997-447f-831e-5906fe8164c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.626490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.608423    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56483207-d997-447f-831e-5906fe8164c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.626490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.626333    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56483207-d997-447f-831e-5906fe8164c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.637460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.637345    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krh5\" (UniqueName: \"kubernetes.io/projected/56483207-d997-447f-831e-5906fe8164c5-kube-api-access-6krh5\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.728578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.728499    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"]
Oct 30 20:25:06.732948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.732926    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.752892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.752863    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"]
Oct 30 20:25:06.785373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.785328    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:06.809918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.809882    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.810144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.810121    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jvs\" (UniqueName: \"kubernetes.io/projected/df6502e0-09df-4f42-bc04-a96b97e60da7-kube-api-access-f8jvs\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.810207 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.810167    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.911589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.911552    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jvs\" (UniqueName: \"kubernetes.io/projected/df6502e0-09df-4f42-bc04-a96b97e60da7-kube-api-access-f8jvs\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.911818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.911607    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.911818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.911696    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.912445 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.912417    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.914279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.914252    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:06.936438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:06.936414    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jvs\" (UniqueName: \"kubernetes.io/projected/df6502e0-09df-4f42-bc04-a96b97e60da7-kube-api-access-f8jvs\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:07.047385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:07.047292    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:07.104000 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:07.103573    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56483207_d997_447f_831e_5906fe8164c5.slice/crio-a52c86179e40089878cab9801dd4a52efddaed305a9bbd435d39c68c03d7fa1a WatchSource:0}: Error finding container a52c86179e40089878cab9801dd4a52efddaed305a9bbd435d39c68c03d7fa1a: Status 404 returned error can't find the container with id a52c86179e40089878cab9801dd4a52efddaed305a9bbd435d39c68c03d7fa1a
Oct 30 20:25:07.104000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:07.103790    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"]
Oct 30 20:25:07.172882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:07.172199    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96" event={"ID":"56483207-d997-447f-831e-5906fe8164c5","Type":"ContainerStarted","Data":"a52c86179e40089878cab9801dd4a52efddaed305a9bbd435d39c68c03d7fa1a"}
Oct 30 20:25:07.367638 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:07.367596    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6502e0_09df_4f42_bc04_a96b97e60da7.slice/crio-2ede04932b165cc6b947660a472a298fca29536fa4b6d4a7071d28b608cc96bb WatchSource:0}: Error finding container 2ede04932b165cc6b947660a472a298fca29536fa4b6d4a7071d28b608cc96bb: Status 404 returned error can't find the container with id 2ede04932b165cc6b947660a472a298fca29536fa4b6d4a7071d28b608cc96bb
Oct 30 20:25:07.368997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:07.368970    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"]
Oct 30 20:25:08.181802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:08.181759    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7" event={"ID":"df6502e0-09df-4f42-bc04-a96b97e60da7","Type":"ContainerStarted","Data":"cc7ed579e71b7c744d4bb010b46466c6610d7a56695227dc7078890b4066c928"}
Oct 30 20:25:08.182243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:08.181826    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7" event={"ID":"df6502e0-09df-4f42-bc04-a96b97e60da7","Type":"ContainerStarted","Data":"2ede04932b165cc6b947660a472a298fca29536fa4b6d4a7071d28b608cc96bb"}
Oct 30 20:25:08.183873 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:08.183845    4519 generic.go:356] "Generic (PLEG): container finished" podID="56483207-d997-447f-831e-5906fe8164c5" containerID="73eeb6a81614e48bd378c527ec04b09703a697978a174d02c48ef96385825b80" exitCode=0
Oct 30 20:25:08.184000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:08.183938    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96" event={"ID":"56483207-d997-447f-831e-5906fe8164c5","Type":"ContainerDied","Data":"73eeb6a81614e48bd378c527ec04b09703a697978a174d02c48ef96385825b80"}
Oct 30 20:25:08.241321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:08.241269    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7" podStartSLOduration=2.241254556 podStartE2EDuration="2.241254556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:08.238812121 +0000 UTC m=+3032.739515845" watchObservedRunningTime="2026-10-30 20:25:08.241254556 +0000 UTC m=+3032.741958338"
Oct 30 20:25:09.134011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.133971    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"]
Oct 30 20:25:09.134255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.134214    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8" podUID="edd58acd-531d-4589-8a3b-b23214045ab4" containerName="discovery" containerID="cri-o://03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0" gracePeriod=30
Oct 30 20:25:09.202298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.202252    4519 generic.go:356] "Generic (PLEG): container finished" podID="df6502e0-09df-4f42-bc04-a96b97e60da7" containerID="cc7ed579e71b7c744d4bb010b46466c6610d7a56695227dc7078890b4066c928" exitCode=0
Oct 30 20:25:09.202759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.202323    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7" event={"ID":"df6502e0-09df-4f42-bc04-a96b97e60da7","Type":"ContainerDied","Data":"cc7ed579e71b7c744d4bb010b46466c6610d7a56695227dc7078890b4066c928"}
Oct 30 20:25:09.378501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.375565    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"]
Oct 30 20:25:09.409766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.409731    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"]
Oct 30 20:25:09.409931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.409867    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.485488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.472019    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.485488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.472177    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4k96\" (UniqueName: \"kubernetes.io/projected/0aade8fd-6341-43f8-9b0e-86353874976a-kube-api-access-j4k96\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.485488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.472221    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.595633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.587691    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.595633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.588481    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.595633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.588756    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4k96\" (UniqueName: \"kubernetes.io/projected/0aade8fd-6341-43f8-9b0e-86353874976a-kube-api-access-j4k96\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.595633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.589120    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.595633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.589795    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:09.604121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.604075    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.617605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.617570    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4k96\" (UniqueName: \"kubernetes.io/projected/0aade8fd-6341-43f8-9b0e-86353874976a-kube-api-access-j4k96\") pod \"maas-api-revoke-sub-3bd42bdb-5xk6w\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.637450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.637423    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 20:25:09.697947 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.697902    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56483207-d997-447f-831e-5906fe8164c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56483207-d997-447f-831e-5906fe8164c5-maas-api-openshift-ca\") pod \"56483207-d997-447f-831e-5906fe8164c5\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") "
Oct 30 20:25:09.698160 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.697992    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56483207-d997-447f-831e-5906fe8164c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56483207-d997-447f-831e-5906fe8164c5-maas-api-xks-ca\") pod \"56483207-d997-447f-831e-5906fe8164c5\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") "
Oct 30 20:25:09.698293 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.698239    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56483207-d997-447f-831e-5906fe8164c5-kube-api-access-6krh5\" (UniqueName: \"kubernetes.io/projected/56483207-d997-447f-831e-5906fe8164c5-kube-api-access-6krh5\") pod \"56483207-d997-447f-831e-5906fe8164c5\" (UID: \"56483207-d997-447f-831e-5906fe8164c5\") "
Oct 30 20:25:09.699597 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.699567    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56483207-d997-447f-831e-5906fe8164c5-maas-api-openshift-ca" pod "56483207-d997-447f-831e-5906fe8164c5" (UID: "56483207-d997-447f-831e-5906fe8164c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:09.700100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.700075    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56483207-d997-447f-831e-5906fe8164c5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.701158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.701130    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56483207-d997-447f-831e-5906fe8164c5-maas-api-xks-ca" pod "56483207-d997-447f-831e-5906fe8164c5" (UID: "56483207-d997-447f-831e-5906fe8164c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:09.701751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.701729    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56483207-d997-447f-831e-5906fe8164c5-kube-api-access-6krh5" pod "56483207-d997-447f-831e-5906fe8164c5" (UID: "56483207-d997-447f-831e-5906fe8164c5"). InnerVolumeSpecName "kube-api-access-6krh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:09.746379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.746347    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:09.801908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.800948    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd58acd-531d-4589-8a3b-b23214045ab4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd58acd-531d-4589-8a3b-b23214045ab4-local-certs\") pod \"edd58acd-531d-4589-8a3b-b23214045ab4\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") "
Oct 30 20:25:09.801908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801012    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-dns-cert\") pod \"edd58acd-531d-4589-8a3b-b23214045ab4\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") "
Oct 30 20:25:09.801908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801146    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-istio-token\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-istio-token\") pod \"edd58acd-531d-4589-8a3b-b23214045ab4\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") "
Oct 30 20:25:09.801908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801227    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-kube-api-access-9f44w\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-kube-api-access-9f44w\") pod \"edd58acd-531d-4589-8a3b-b23214045ab4\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") "
Oct 30 20:25:09.801908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801328    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-ca-configmap\") pod \"edd58acd-531d-4589-8a3b-b23214045ab4\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") "
Oct 30 20:25:09.801908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801439    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-kubeconfig\") pod \"edd58acd-531d-4589-8a3b-b23214045ab4\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") "
Oct 30 20:25:09.801908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801505    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-cacerts\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-cacerts\") pod \"edd58acd-531d-4589-8a3b-b23214045ab4\" (UID: \"edd58acd-531d-4589-8a3b-b23214045ab4\") "
Oct 30 20:25:09.802740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801927    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krh5\" (UniqueName: \"kubernetes.io/projected/56483207-d997-447f-831e-5906fe8164c5-kube-api-access-6krh5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.802740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.801940    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56483207-d997-447f-831e-5906fe8164c5-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.802740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.802724    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-ca-configmap" pod "edd58acd-531d-4589-8a3b-b23214045ab4" (UID: "edd58acd-531d-4589-8a3b-b23214045ab4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:09.806154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.805683    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-kubeconfig" pod "edd58acd-531d-4589-8a3b-b23214045ab4" (UID: "edd58acd-531d-4589-8a3b-b23214045ab4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:09.806154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.805762    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-istio-token" pod "edd58acd-531d-4589-8a3b-b23214045ab4" (UID: "edd58acd-531d-4589-8a3b-b23214045ab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:09.806154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.805896    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd58acd-531d-4589-8a3b-b23214045ab4-local-certs" pod "edd58acd-531d-4589-8a3b-b23214045ab4" (UID: "edd58acd-531d-4589-8a3b-b23214045ab4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:09.806154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.806105    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-cacerts" pod "edd58acd-531d-4589-8a3b-b23214045ab4" (UID: "edd58acd-531d-4589-8a3b-b23214045ab4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:09.806379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.806175    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-dns-cert" pod "edd58acd-531d-4589-8a3b-b23214045ab4" (UID: "edd58acd-531d-4589-8a3b-b23214045ab4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:09.807094 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.807074    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-kube-api-access-9f44w" pod "edd58acd-531d-4589-8a3b-b23214045ab4" (UID: "edd58acd-531d-4589-8a3b-b23214045ab4"). InnerVolumeSpecName "kube-api-access-9f44w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:09.903368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.903327    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.903368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.903363    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f44w\" (UniqueName: \"kubernetes.io/projected/edd58acd-531d-4589-8a3b-b23214045ab4-kube-api-access-9f44w\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.903368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.903376    4519 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-ca-configmap\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.903686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.903386    4519 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-kubeconfig\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.903686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.903395    4519 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-cacerts\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.903686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.903403    4519 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd58acd-531d-4589-8a3b-b23214045ab4-local-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:09.903686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:09.903411    4519 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/edd58acd-531d-4589-8a3b-b23214045ab4-istio-csr-dns-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:10.072515 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:10.072442    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aade8fd_6341_43f8_9b0e_86353874976a.slice/crio-65fa43df98be6f26911f45550386e049b4923105a23f79efe8dba5587945c1c6 WatchSource:0}: Error finding container 65fa43df98be6f26911f45550386e049b4923105a23f79efe8dba5587945c1c6: Status 404 returned error can't find the container with id 65fa43df98be6f26911f45550386e049b4923105a23f79efe8dba5587945c1c6
Oct 30 20:25:10.074851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.074784    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"]
Oct 30 20:25:10.212585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.212551    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"
Oct 30 20:25:10.212585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.212566    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96" event={"ID":"56483207-d997-447f-831e-5906fe8164c5","Type":"ContainerDied","Data":"a52c86179e40089878cab9801dd4a52efddaed305a9bbd435d39c68c03d7fa1a"}
Oct 30 20:25:10.213083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.212631    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a52c86179e40089878cab9801dd4a52efddaed305a9bbd435d39c68c03d7fa1a"
Oct 30 20:25:10.214437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.214401    4519 generic.go:356] "Generic (PLEG): container finished" podID="edd58acd-531d-4589-8a3b-b23214045ab4" containerID="03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0" exitCode=0
Oct 30 20:25:10.214631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.214442    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8" event={"ID":"edd58acd-531d-4589-8a3b-b23214045ab4","Type":"ContainerDied","Data":"03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0"}
Oct 30 20:25:10.214631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.214488    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"
Oct 30 20:25:10.214631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.214539    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8" event={"ID":"edd58acd-531d-4589-8a3b-b23214045ab4","Type":"ContainerDied","Data":"7c1915941752652f691f01282fbd19923c85f4e7999c42dc9ad1f7793bc753d4"}
Oct 30 20:25:10.214631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.214581    4519 scope.go:122] "RemoveContainer" containerID="03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0"
Oct 30 20:25:10.220124 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.220096    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w" event={"ID":"0aade8fd-6341-43f8-9b0e-86353874976a","Type":"ContainerStarted","Data":"65fa43df98be6f26911f45550386e049b4923105a23f79efe8dba5587945c1c6"}
Oct 30 20:25:10.296259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.296139    4519 scope.go:122] "RemoveContainer" containerID="03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0"
Oct 30 20:25:10.296898 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:10.296866    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0\": container with ID starting with 03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0 not found: ID does not exist" containerID="03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0"
Oct 30 20:25:10.297009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.296912    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0"} err="failed to get container status \"03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0\": rpc error: code = NotFound desc = could not find container \"03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0\": container with ID starting with 03dd762c76a033b45d52d29fc9dbcf8fb99fbd4e2808b743e6a8f999024ea3d0 not found: ID does not exist"
Oct 30 20:25:10.342772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.342695    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"]
Oct 30 20:25:10.363354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.363319    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk4j8"]
Oct 30 20:25:10.417996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.417966    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:10.512177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.512134    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-xks-ca\") pod \"df6502e0-09df-4f42-bc04-a96b97e60da7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") "
Oct 30 20:25:10.512391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.512370    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6502e0-09df-4f42-bc04-a96b97e60da7-kube-api-access-f8jvs\" (UniqueName: \"kubernetes.io/projected/df6502e0-09df-4f42-bc04-a96b97e60da7-kube-api-access-f8jvs\") pod \"df6502e0-09df-4f42-bc04-a96b97e60da7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") "
Oct 30 20:25:10.512595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.512565    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-openshift-ca\") pod \"df6502e0-09df-4f42-bc04-a96b97e60da7\" (UID: \"df6502e0-09df-4f42-bc04-a96b97e60da7\") "
Oct 30 20:25:10.513025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.513000    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-openshift-ca" pod "df6502e0-09df-4f42-bc04-a96b97e60da7" (UID: "df6502e0-09df-4f42-bc04-a96b97e60da7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:10.513203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.513185    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:10.514360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.514340    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-xks-ca" pod "df6502e0-09df-4f42-bc04-a96b97e60da7" (UID: "df6502e0-09df-4f42-bc04-a96b97e60da7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:10.515004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.514980    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6502e0-09df-4f42-bc04-a96b97e60da7-kube-api-access-f8jvs" pod "df6502e0-09df-4f42-bc04-a96b97e60da7" (UID: "df6502e0-09df-4f42-bc04-a96b97e60da7"). InnerVolumeSpecName "kube-api-access-f8jvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:10.614881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.614798    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8jvs\" (UniqueName: \"kubernetes.io/projected/df6502e0-09df-4f42-bc04-a96b97e60da7-kube-api-access-f8jvs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:10.614881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.614830    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6502e0-09df-4f42-bc04-a96b97e60da7-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:10.922617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.922584    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"]
Oct 30 20:25:10.923182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923163    4519 cpu_manager.go:414] "containerMap: removing container" podUID="edd58acd-531d-4589-8a3b-b23214045ab4" containerName="discovery"
Oct 30 20:25:10.923182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923179    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd58acd-531d-4589-8a3b-b23214045ab4" containerName="discovery"
Oct 30 20:25:10.923344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923192    4519 cpu_manager.go:414] "containerMap: removing container" podUID="df6502e0-09df-4f42-bc04-a96b97e60da7" containerName="revoke-keys"
Oct 30 20:25:10.923344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923198    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6502e0-09df-4f42-bc04-a96b97e60da7" containerName="revoke-keys"
Oct 30 20:25:10.923344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923226    4519 cpu_manager.go:414] "containerMap: removing container" podUID="56483207-d997-447f-831e-5906fe8164c5" containerName="revoke-keys"
Oct 30 20:25:10.923344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923231    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56483207-d997-447f-831e-5906fe8164c5" containerName="revoke-keys"
Oct 30 20:25:10.923344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923317    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="56483207-d997-447f-831e-5906fe8164c5" containerName="revoke-keys"
Oct 30 20:25:10.923344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923331    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd58acd-531d-4589-8a3b-b23214045ab4" containerName="discovery"
Oct 30 20:25:10.923344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.923337    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6502e0-09df-4f42-bc04-a96b97e60da7" containerName="revoke-keys"
Oct 30 20:25:10.927113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.927092    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:10.929119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.929100    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 20:25:10.951523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:10.949385    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"]
Oct 30 20:25:11.022204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.022158    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-data" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:11.022409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.022233    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-data\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.023554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.023524    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-podinfo\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.024184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024160    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-certs\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.024409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024389    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-certs" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:11.024582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024554    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-socket\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.024697 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024629    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-credential-socket\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.025093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024886    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-token\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-token\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.025093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024905    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-credential-socket" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:11.025093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024982    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-envoy\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.025093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.024988    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-socket" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:11.025093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025040    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-kube-api-access-wrjw5\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-kube-api-access-wrjw5\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.025400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025153    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istiod-ca-cert\") pod \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\" (UID: \"e39e72f2-f9c1-4d6a-b581-99d1a2555536\") "
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025458    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhjp\" (UniqueName: \"kubernetes.io/projected/a3c4fb3d-0052-422f-86db-92f5a57fac28-kube-api-access-zvhjp\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025558    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025662    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025961    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025982    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.025999    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.026012    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.026101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.026023    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istiod-ca-cert" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:11.027607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.027580    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-podinfo" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:11.028424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.028395    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-kube-api-access-wrjw5" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "kube-api-access-wrjw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:11.028643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.028624    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-envoy" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:11.028758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.028742    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-token" pod "e39e72f2-f9c1-4d6a-b581-99d1a2555536" (UID: "e39e72f2-f9c1-4d6a-b581-99d1a2555536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:11.127039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127007    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhjp\" (UniqueName: \"kubernetes.io/projected/a3c4fb3d-0052-422f-86db-92f5a57fac28-kube-api-access-zvhjp\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.127232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127059    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.127232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127122    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.127357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127293    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.127357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127313    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.127357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127327    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjw5\" (UniqueName: \"kubernetes.io/projected/e39e72f2-f9c1-4d6a-b581-99d1a2555536-kube-api-access-wrjw5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.127357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127345    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.127590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127359    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e72f2-f9c1-4d6a-b581-99d1a2555536-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:11.127850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.127828    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.129636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.129610    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.148174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.148141    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"]
Oct 30 20:25:11.152387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.152357    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhjp\" (UniqueName: \"kubernetes.io/projected/a3c4fb3d-0052-422f-86db-92f5a57fac28-kube-api-access-zvhjp\") pod \"maas-api-revoke-sub-d130d9ef-rplcf\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.240559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.240517    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:11.241201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.241171    4519 generic.go:356] "Generic (PLEG): container finished" podID="0aade8fd-6341-43f8-9b0e-86353874976a" containerID="8d1910a99e5f0a398e91fa5a0bec8b8363b16a7cec10c99d1bf60455e88e2403" exitCode=0
Oct 30 20:25:11.241291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.241251    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w" event={"ID":"0aade8fd-6341-43f8-9b0e-86353874976a","Type":"ContainerDied","Data":"8d1910a99e5f0a398e91fa5a0bec8b8363b16a7cec10c99d1bf60455e88e2403"}
Oct 30 20:25:11.242880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.242854    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"
Oct 30 20:25:11.243009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.242871    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7" event={"ID":"df6502e0-09df-4f42-bc04-a96b97e60da7","Type":"ContainerDied","Data":"2ede04932b165cc6b947660a472a298fca29536fa4b6d4a7071d28b608cc96bb"}
Oct 30 20:25:11.243009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.242917    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ede04932b165cc6b947660a472a298fca29536fa4b6d4a7071d28b608cc96bb"
Oct 30 20:25:11.245057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.245032    4519 generic.go:356] "Generic (PLEG): container finished" podID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerID="64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b" exitCode=0
Oct 30 20:25:11.245161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.245083    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"
Oct 30 20:25:11.245161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.245103    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" event={"ID":"e39e72f2-f9c1-4d6a-b581-99d1a2555536","Type":"ContainerDied","Data":"64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b"}
Oct 30 20:25:11.245161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.245146    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb" event={"ID":"e39e72f2-f9c1-4d6a-b581-99d1a2555536","Type":"ContainerDied","Data":"51e5ec7b9357660501c420f4384963c23ff7fee9e2de614a939dca4bb943e71b"}
Oct 30 20:25:11.245426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.245170    4519 scope.go:122] "RemoveContainer" containerID="64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b"
Oct 30 20:25:11.296856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.296826    4519 scope.go:122] "RemoveContainer" containerID="ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac"
Oct 30 20:25:11.311622 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.311600    4519 scope.go:122] "RemoveContainer" containerID="64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b"
Oct 30 20:25:11.312455 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:11.312428    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b\": container with ID starting with 64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b not found: ID does not exist" containerID="64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b"
Oct 30 20:25:11.312455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.312484    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b"} err="failed to get container status \"64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b\": rpc error: code = NotFound desc = could not find container \"64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b\": container with ID starting with 64ba0623e639480e7b2659c7ba14b0353cc0dca5bd6e7cbb0fd41f784ffb522b not found: ID does not exist"
Oct 30 20:25:11.312970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.312509    4519 scope.go:122] "RemoveContainer" containerID="ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac"
Oct 30 20:25:11.313652 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:11.313628    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac\": container with ID starting with ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac not found: ID does not exist" containerID="ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac"
Oct 30 20:25:11.313747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.313660    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac"} err="failed to get container status \"ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac\": rpc error: code = NotFound desc = could not find container \"ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac\": container with ID starting with ae305ce0ad04f5f9f26ed0fd74c9418905894c231c44efa9e844e02c7f76a7ac not found: ID does not exist"
Oct 30 20:25:11.339431 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:11.338695    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39e72f2_f9c1_4d6a_b581_99d1a2555536.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39e72f2_f9c1_4d6a_b581_99d1a2555536.slice/crio-51e5ec7b9357660501c420f4384963c23ff7fee9e2de614a939dca4bb943e71b\": RecentStats: unable to find data in memory cache]"
Oct 30 20:25:11.361308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.361280    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"]
Oct 30 20:25:11.419062 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.419022    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-mhchb"]
Oct 30 20:25:11.614297 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:11.614258    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c4fb3d_0052_422f_86db_92f5a57fac28.slice/crio-49438f2c07028fd62582f7c6641c703b5e504b685b17a5a6439abdfb8cde023e WatchSource:0}: Error finding container 49438f2c07028fd62582f7c6641c703b5e504b685b17a5a6439abdfb8cde023e: Status 404 returned error can't find the container with id 49438f2c07028fd62582f7c6641c703b5e504b685b17a5a6439abdfb8cde023e
Oct 30 20:25:11.616558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.616527    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" path="/var/lib/kubelet/pods/e39e72f2-f9c1-4d6a-b581-99d1a2555536/volumes"
Oct 30 20:25:11.617231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.617212    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd58acd-531d-4589-8a3b-b23214045ab4" path="/var/lib/kubelet/pods/edd58acd-531d-4589-8a3b-b23214045ab4/volumes"
Oct 30 20:25:11.617682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:11.617661    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"]
Oct 30 20:25:12.256993 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.256954    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf" event={"ID":"a3c4fb3d-0052-422f-86db-92f5a57fac28","Type":"ContainerStarted","Data":"9700f8c7f2a2ef22ccf39c1a1e091c6bafe0ed19eca6d8e631598db708767693"}
Oct 30 20:25:12.257535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.257017    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf" event={"ID":"a3c4fb3d-0052-422f-86db-92f5a57fac28","Type":"ContainerStarted","Data":"49438f2c07028fd62582f7c6641c703b5e504b685b17a5a6439abdfb8cde023e"}
Oct 30 20:25:12.316121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.316065    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf" podStartSLOduration=2.316049223 podStartE2EDuration="2.316049223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:12.312014433 +0000 UTC m=+3036.812718158" watchObservedRunningTime="2026-10-30 20:25:12.316049223 +0000 UTC m=+3036.816752943"
Oct 30 20:25:12.441403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.441377    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:12.551651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.551612    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aade8fd-6341-43f8-9b0e-86353874976a-kube-api-access-j4k96\" (UniqueName: \"kubernetes.io/projected/0aade8fd-6341-43f8-9b0e-86353874976a-kube-api-access-j4k96\") pod \"0aade8fd-6341-43f8-9b0e-86353874976a\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") "
Oct 30 20:25:12.551843 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.551822    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-xks-ca\") pod \"0aade8fd-6341-43f8-9b0e-86353874976a\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") "
Oct 30 20:25:12.551985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.551968    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-openshift-ca\") pod \"0aade8fd-6341-43f8-9b0e-86353874976a\" (UID: \"0aade8fd-6341-43f8-9b0e-86353874976a\") "
Oct 30 20:25:12.552437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.552411    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-openshift-ca" pod "0aade8fd-6341-43f8-9b0e-86353874976a" (UID: "0aade8fd-6341-43f8-9b0e-86353874976a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:12.552642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.552620    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:12.554666 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.554633    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aade8fd-6341-43f8-9b0e-86353874976a-kube-api-access-j4k96" pod "0aade8fd-6341-43f8-9b0e-86353874976a" (UID: "0aade8fd-6341-43f8-9b0e-86353874976a"). InnerVolumeSpecName "kube-api-access-j4k96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:12.554763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.554715    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-xks-ca" pod "0aade8fd-6341-43f8-9b0e-86353874976a" (UID: "0aade8fd-6341-43f8-9b0e-86353874976a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:12.653654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.653622    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4k96\" (UniqueName: \"kubernetes.io/projected/0aade8fd-6341-43f8-9b0e-86353874976a-kube-api-access-j4k96\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:12.653654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:12.653655    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aade8fd-6341-43f8-9b0e-86353874976a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:13.268424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:13.268395    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"
Oct 30 20:25:13.268424 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:13.268413    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w" event={"ID":"0aade8fd-6341-43f8-9b0e-86353874976a","Type":"ContainerDied","Data":"65fa43df98be6f26911f45550386e049b4923105a23f79efe8dba5587945c1c6"}
Oct 30 20:25:13.269039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:13.268487    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65fa43df98be6f26911f45550386e049b4923105a23f79efe8dba5587945c1c6"
Oct 30 20:25:13.270429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:13.270396    4519 generic.go:356] "Generic (PLEG): container finished" podID="a3c4fb3d-0052-422f-86db-92f5a57fac28" containerID="9700f8c7f2a2ef22ccf39c1a1e091c6bafe0ed19eca6d8e631598db708767693" exitCode=0
Oct 30 20:25:13.270603 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:13.270511    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf" event={"ID":"a3c4fb3d-0052-422f-86db-92f5a57fac28","Type":"ContainerDied","Data":"9700f8c7f2a2ef22ccf39c1a1e091c6bafe0ed19eca6d8e631598db708767693"}
Oct 30 20:25:14.447872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.447845    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:14.501545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.501509    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-xks-ca\") pod \"a3c4fb3d-0052-422f-86db-92f5a57fac28\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") "
Oct 30 20:25:14.501720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.501582    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-openshift-ca\") pod \"a3c4fb3d-0052-422f-86db-92f5a57fac28\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") "
Oct 30 20:25:14.502318 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.502285    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-openshift-ca" pod "a3c4fb3d-0052-422f-86db-92f5a57fac28" (UID: "a3c4fb3d-0052-422f-86db-92f5a57fac28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:14.504733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.504701    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-xks-ca" pod "a3c4fb3d-0052-422f-86db-92f5a57fac28" (UID: "a3c4fb3d-0052-422f-86db-92f5a57fac28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:14.603306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.603199    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c4fb3d-0052-422f-86db-92f5a57fac28-kube-api-access-zvhjp\" (UniqueName: \"kubernetes.io/projected/a3c4fb3d-0052-422f-86db-92f5a57fac28-kube-api-access-zvhjp\") pod \"a3c4fb3d-0052-422f-86db-92f5a57fac28\" (UID: \"a3c4fb3d-0052-422f-86db-92f5a57fac28\") "
Oct 30 20:25:14.603724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.603703    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:14.603823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.603728    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4fb3d-0052-422f-86db-92f5a57fac28-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:14.605335 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.605307    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c4fb3d-0052-422f-86db-92f5a57fac28-kube-api-access-zvhjp" pod "a3c4fb3d-0052-422f-86db-92f5a57fac28" (UID: "a3c4fb3d-0052-422f-86db-92f5a57fac28"). InnerVolumeSpecName "kube-api-access-zvhjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:14.705253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:14.705213    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhjp\" (UniqueName: \"kubernetes.io/projected/a3c4fb3d-0052-422f-86db-92f5a57fac28-kube-api-access-zvhjp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:15.292385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:15.292354    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"
Oct 30 20:25:15.292571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:15.292384    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf" event={"ID":"a3c4fb3d-0052-422f-86db-92f5a57fac28","Type":"ContainerDied","Data":"49438f2c07028fd62582f7c6641c703b5e504b685b17a5a6439abdfb8cde023e"}
Oct 30 20:25:15.292571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:15.292448    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49438f2c07028fd62582f7c6641c703b5e504b685b17a5a6439abdfb8cde023e"
Oct 30 20:25:16.041535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:16.039930    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"]
Oct 30 20:25:16.063487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:16.060922    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673dc401-w2zkj"]
Oct 30 20:25:17.173262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:17.173225    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"]
Oct 30 20:25:17.613459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:17.613375    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d136ce1-3be3-44ed-a735-0c02b77ccc05" path="/var/lib/kubelet/pods/0d136ce1-3be3-44ed-a735-0c02b77ccc05/volumes"
Oct 30 20:25:20.943454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:20.943405    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"]
Oct 30 20:25:20.943953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:20.943697    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6" podUID="99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" containerName="payload-processing" containerID="cri-o://a697156ae662aa56638003b418b5b0c727d663ef735a34e6dadb35e4a786ccbc" gracePeriod=30
Oct 30 20:25:21.475501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.471446    4519 generic.go:356] "Generic (PLEG): container finished" podID="99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" containerID="a697156ae662aa56638003b418b5b0c727d663ef735a34e6dadb35e4a786ccbc" exitCode=0
Oct 30 20:25:21.475501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.471668    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6" event={"ID":"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac","Type":"ContainerDied","Data":"a697156ae662aa56638003b418b5b0c727d663ef735a34e6dadb35e4a786ccbc"}
Oct 30 20:25:21.532762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.532735    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:25:21.694136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.688871    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-plugins-config-volume\") pod \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") "
Oct 30 20:25:21.694136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.688985    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-kube-api-access-f8nzc\" (UniqueName: \"kubernetes.io/projected/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-kube-api-access-f8nzc\") pod \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\" (UID: \"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac\") "
Oct 30 20:25:21.694136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.691786    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-plugins-config-volume" pod "99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" (UID: "99138e4e-7ff6-4cbf-873b-b0ec0d5eefac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:21.694635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.694599    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-kube-api-access-f8nzc" pod "99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" (UID: "99138e4e-7ff6-4cbf-873b-b0ec0d5eefac"). InnerVolumeSpecName "kube-api-access-f8nzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:21.793096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.792995    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8nzc\" (UniqueName: \"kubernetes.io/projected/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-kube-api-access-f8nzc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:21.793330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.793306    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:21.887492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.887305    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"]
Oct 30 20:25:21.894884 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:21.894844    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7eec8-25cq7"]
Oct 30 20:25:22.174016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:22.173970    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerName="payload-pre-processing" containerID="cri-o://05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774" gracePeriod=25
Oct 30 20:25:22.441994 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:22.441881    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:25:22.481956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:22.481930    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"
Oct 30 20:25:22.482136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:22.481967    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6" event={"ID":"99138e4e-7ff6-4cbf-873b-b0ec0d5eefac","Type":"ContainerDied","Data":"c515a727c45f458525f4b08692db5adb1b557e4d68736bf0266c52d284e9924d"}
Oct 30 20:25:22.482136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:22.482036    4519 scope.go:122] "RemoveContainer" containerID="a697156ae662aa56638003b418b5b0c727d663ef735a34e6dadb35e4a786ccbc"
Oct 30 20:25:22.539674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:22.539637    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"]
Oct 30 20:25:22.549228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:22.549188    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-747d98db9b-n2vx6"]
Oct 30 20:25:23.612968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:23.612928    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" path="/var/lib/kubelet/pods/99138e4e-7ff6-4cbf-873b-b0ec0d5eefac/volumes"
Oct 30 20:25:23.613491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:23.613447    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6502e0-09df-4f42-bc04-a96b97e60da7" path="/var/lib/kubelet/pods/df6502e0-09df-4f42-bc04-a96b97e60da7/volumes"
Oct 30 20:25:26.038208 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.038168    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"]
Oct 30 20:25:26.048844 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.048812    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0b9bc1-xhcsb"]
Oct 30 20:25:26.171781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.171722    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main" containerID="cri-o://8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b" gracePeriod=45
Oct 30 20:25:26.442643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.442612    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 20:25:26.465280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465246    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-model-cache\") pod \"f02d3d7d-3a67-4934-995f-12773e9a5874\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") "
Oct 30 20:25:26.465484 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465313    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-tmp-dir\") pod \"f02d3d7d-3a67-4934-995f-12773e9a5874\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") "
Oct 30 20:25:26.465568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465483    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02d3d7d-3a67-4934-995f-12773e9a5874-kube-api-access-cw2jj\" (UniqueName: \"kubernetes.io/projected/f02d3d7d-3a67-4934-995f-12773e9a5874-kube-api-access-cw2jj\") pod \"f02d3d7d-3a67-4934-995f-12773e9a5874\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") "
Oct 30 20:25:26.465568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465542    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02d3d7d-3a67-4934-995f-12773e9a5874-tls-certs\" (UniqueName: \"kubernetes.io/secret/f02d3d7d-3a67-4934-995f-12773e9a5874-tls-certs\") pod \"f02d3d7d-3a67-4934-995f-12773e9a5874\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") "
Oct 30 20:25:26.465700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465606    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-dshm\") pod \"f02d3d7d-3a67-4934-995f-12773e9a5874\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") "
Oct 30 20:25:26.465700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465596    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-model-cache" pod "f02d3d7d-3a67-4934-995f-12773e9a5874" (UID: "f02d3d7d-3a67-4934-995f-12773e9a5874"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:26.465700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465637    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-home\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-home\") pod \"f02d3d7d-3a67-4934-995f-12773e9a5874\" (UID: \"f02d3d7d-3a67-4934-995f-12773e9a5874\") "
Oct 30 20:25:26.465700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465626    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-tmp-dir" pod "f02d3d7d-3a67-4934-995f-12773e9a5874" (UID: "f02d3d7d-3a67-4934-995f-12773e9a5874"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:26.465975 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.465951    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-home" pod "f02d3d7d-3a67-4934-995f-12773e9a5874" (UID: "f02d3d7d-3a67-4934-995f-12773e9a5874"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:26.466196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.466176    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:26.466315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.466198    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:26.466315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.466213    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:26.467938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.467907    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02d3d7d-3a67-4934-995f-12773e9a5874-tls-certs" pod "f02d3d7d-3a67-4934-995f-12773e9a5874" (UID: "f02d3d7d-3a67-4934-995f-12773e9a5874"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:26.468232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.468207    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-dshm" pod "f02d3d7d-3a67-4934-995f-12773e9a5874" (UID: "f02d3d7d-3a67-4934-995f-12773e9a5874"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:26.468337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.468318    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02d3d7d-3a67-4934-995f-12773e9a5874-kube-api-access-cw2jj" pod "f02d3d7d-3a67-4934-995f-12773e9a5874" (UID: "f02d3d7d-3a67-4934-995f-12773e9a5874"). InnerVolumeSpecName "kube-api-access-cw2jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:26.524717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.523943    4519 generic.go:356] "Generic (PLEG): container finished" podID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerID="8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b" exitCode=0
Oct 30 20:25:26.524717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.524100    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" event={"ID":"f02d3d7d-3a67-4934-995f-12773e9a5874","Type":"ContainerDied","Data":"8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b"}
Oct 30 20:25:26.524717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.524190    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc" event={"ID":"f02d3d7d-3a67-4934-995f-12773e9a5874","Type":"ContainerDied","Data":"520acbf5888bc55e3f0f5479135a9089ac2279566a37cccf818ba49258daadde"}
Oct 30 20:25:26.524717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.524258    4519 scope.go:122] "RemoveContainer" containerID="8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b"
Oct 30 20:25:26.524717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.524549    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"
Oct 30 20:25:26.555573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.555551    4519 scope.go:122] "RemoveContainer" containerID="8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b"
Oct 30 20:25:26.556300 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:26.556267    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b\": container with ID starting with 8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b not found: ID does not exist" containerID="8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b"
Oct 30 20:25:26.556428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.556310    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b"} err="failed to get container status \"8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b\": rpc error: code = NotFound desc = could not find container \"8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b\": container with ID starting with 8f510072587a932d9aa0a324c04ca4996afaf0dfcc4d0a537179534a8e8e051b not found: ID does not exist"
Oct 30 20:25:26.567862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.567831    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw2jj\" (UniqueName: \"kubernetes.io/projected/f02d3d7d-3a67-4934-995f-12773e9a5874-kube-api-access-cw2jj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:26.567992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.567866    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02d3d7d-3a67-4934-995f-12773e9a5874-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:26.567992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.567884    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02d3d7d-3a67-4934-995f-12773e9a5874-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:26.587671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.587629    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"]
Oct 30 20:25:26.592417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:26.592387    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/simulator-w4-8a3426-kserve-6c4cfd45c9-6b2hc"]
Oct 30 20:25:27.632490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:27.628525    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e28ee8c-8795-4313-8b01-92845e6c40e3" path="/var/lib/kubelet/pods/8e28ee8c-8795-4313-8b01-92845e6c40e3/volumes"
Oct 30 20:25:27.632490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:27.629988    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" path="/var/lib/kubelet/pods/f02d3d7d-3a67-4934-995f-12773e9a5874/volumes"
Oct 30 20:25:29.245995 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:29.245956    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"]
Oct 30 20:25:29.246645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:29.246297    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" containerID="cri-o://bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b" gracePeriod=30
Oct 30 20:25:30.439643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.439599    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"]
Oct 30 20:25:30.440514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440453    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a3c4fb3d-0052-422f-86db-92f5a57fac28" containerName="revoke-keys"
Oct 30 20:25:30.440514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440492    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c4fb3d-0052-422f-86db-92f5a57fac28" containerName="revoke-keys"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440524    4519 cpu_manager.go:414] "containerMap: removing container" podUID="99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" containerName="payload-processing"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440532    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" containerName="payload-processing"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440547    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440555    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440568    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0aade8fd-6341-43f8-9b0e-86353874976a" containerName="revoke-keys"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440576    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aade8fd-6341-43f8-9b0e-86353874976a" containerName="revoke-keys"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440611    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy"
Oct 30 20:25:30.440656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440618    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy"
Oct 30 20:25:30.440916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440745    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy"
Oct 30 20:25:30.440916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440758    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy"
Oct 30 20:25:30.440916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440772    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c4fb3d-0052-422f-86db-92f5a57fac28" containerName="revoke-keys"
Oct 30 20:25:30.440916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440784    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="99138e4e-7ff6-4cbf-873b-b0ec0d5eefac" containerName="payload-processing"
Oct 30 20:25:30.440916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440792    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02d3d7d-3a67-4934-995f-12773e9a5874" containerName="main"
Oct 30 20:25:30.440916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.440799    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aade8fd-6341-43f8-9b0e-86353874976a" containerName="revoke-keys"
Oct 30 20:25:30.446265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.446241    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.469960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.469919    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"]
Oct 30 20:25:30.475958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.475924    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:25:30.476118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.476044    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:25:30.573902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.570302    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnclk\" (UniqueName: \"kubernetes.io/projected/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-kube-api-access-dnclk\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.573902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.570399    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.573902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.570440    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.671459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.671407    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnclk\" (UniqueName: \"kubernetes.io/projected/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-kube-api-access-dnclk\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.671697 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.671528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.671697 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.671568    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.672410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.672384    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.674352 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.674324    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.710529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.710434    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnclk\" (UniqueName: \"kubernetes.io/projected/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-kube-api-access-dnclk\") pod \"maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:30.764436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:30.764398    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:31.118599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:31.118556    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"]
Oct 30 20:25:31.599270 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:31.599226    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv" event={"ID":"dd74ffd0-9048-4dac-96b2-d14f5ba3703a","Type":"ContainerStarted","Data":"2d31ef180fc2b58f796bb177704c08f2d2d2159558cfe64d10bb9d256befff86"}
Oct 30 20:25:31.599747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:31.599309    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv" event={"ID":"dd74ffd0-9048-4dac-96b2-d14f5ba3703a","Type":"ContainerStarted","Data":"76caf4de689133284bcaeed8ff2ba22922f9461cdd6557a7aaf5023607abeaf2"}
Oct 30 20:25:31.645134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:31.645080    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv" podStartSLOduration=1.645065488 podStartE2EDuration="1.645065488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:31.64090301 +0000 UTC m=+3056.141606734" watchObservedRunningTime="2026-10-30 20:25:31.645065488 +0000 UTC m=+3056.145769212"
Oct 30 20:25:32.222346 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:32.222290    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:25:32.406046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:32.405998    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:25:32.406253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:32.406067    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:25:32.441319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:32.441218    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:25:32.610602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:32.610565    4519 generic.go:356] "Generic (PLEG): container finished" podID="dd74ffd0-9048-4dac-96b2-d14f5ba3703a" containerID="2d31ef180fc2b58f796bb177704c08f2d2d2159558cfe64d10bb9d256befff86" exitCode=0
Oct 30 20:25:32.611102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:32.610641    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv" event={"ID":"dd74ffd0-9048-4dac-96b2-d14f5ba3703a","Type":"ContainerDied","Data":"2d31ef180fc2b58f796bb177704c08f2d2d2159558cfe64d10bb9d256befff86"}
Oct 30 20:25:33.825664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.825633    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:33.930934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.930893    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-openshift-ca\") pod \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") "
Oct 30 20:25:33.931159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.931010    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-xks-ca\") pod \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") "
Oct 30 20:25:33.931159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.931067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-kube-api-access-dnclk\" (UniqueName: \"kubernetes.io/projected/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-kube-api-access-dnclk\") pod \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\" (UID: \"dd74ffd0-9048-4dac-96b2-d14f5ba3703a\") "
Oct 30 20:25:33.931404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.931371    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-openshift-ca" pod "dd74ffd0-9048-4dac-96b2-d14f5ba3703a" (UID: "dd74ffd0-9048-4dac-96b2-d14f5ba3703a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:33.931710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.931691    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:33.933510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.933460    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-kube-api-access-dnclk" pod "dd74ffd0-9048-4dac-96b2-d14f5ba3703a" (UID: "dd74ffd0-9048-4dac-96b2-d14f5ba3703a"). InnerVolumeSpecName "kube-api-access-dnclk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:33.933619 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:33.933543    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-xks-ca" pod "dd74ffd0-9048-4dac-96b2-d14f5ba3703a" (UID: "dd74ffd0-9048-4dac-96b2-d14f5ba3703a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:34.032830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.032798    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.032830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.032825    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnclk\" (UniqueName: \"kubernetes.io/projected/dd74ffd0-9048-4dac-96b2-d14f5ba3703a-kube-api-access-dnclk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.531755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.531685    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:25:34.633969 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.633933    4519 generic.go:356] "Generic (PLEG): container finished" podID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerID="bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b" exitCode=0
Oct 30 20:25:34.634175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.633995    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"
Oct 30 20:25:34.634175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.634018    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" event={"ID":"26412140-b8b8-428b-802f-4e1a66f8fd79","Type":"ContainerDied","Data":"bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b"}
Oct 30 20:25:34.634175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.634082    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4" event={"ID":"26412140-b8b8-428b-802f-4e1a66f8fd79","Type":"ContainerDied","Data":"ab23335e3b6d7738d1b9e82ca94d2e3695918dd546d0d976fcd69a75432eda86"}
Oct 30 20:25:34.634175 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.634117    4519 scope.go:122] "RemoveContainer" containerID="bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b"
Oct 30 20:25:34.636317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.636286    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"
Oct 30 20:25:34.636414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.636310    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv" event={"ID":"dd74ffd0-9048-4dac-96b2-d14f5ba3703a","Type":"ContainerDied","Data":"76caf4de689133284bcaeed8ff2ba22922f9461cdd6557a7aaf5023607abeaf2"}
Oct 30 20:25:34.636414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.636360    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76caf4de689133284bcaeed8ff2ba22922f9461cdd6557a7aaf5023607abeaf2"
Oct 30 20:25:34.640527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640505    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26412140-b8b8-428b-802f-4e1a66f8fd79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26412140-b8b8-428b-802f-4e1a66f8fd79-istiod-ca-cert\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.640635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-socket\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.640635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640606    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-certs\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.640733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640658    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-data\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.640733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640698    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-envoy\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.640851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640761    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-kube-api-access-zfc2f\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-kube-api-access-zfc2f\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.640916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640869    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-socket" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.640916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640868    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26412140-b8b8-428b-802f-4e1a66f8fd79-istiod-ca-cert" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:34.641028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.640986    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-certs" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.641072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641018    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-data" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.641072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641032    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-podinfo\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.641188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641077    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-credential-socket\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.641188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641115    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-token\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-token\") pod \"26412140-b8b8-428b-802f-4e1a66f8fd79\" (UID: \"26412140-b8b8-428b-802f-4e1a66f8fd79\") "
Oct 30 20:25:34.641436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641411    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-credential-socket" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.641758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641736    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.641866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641765    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26412140-b8b8-428b-802f-4e1a66f8fd79-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.641866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641781    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.641866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641794    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.641866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.641808    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.643566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.643537    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-envoy" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.643665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.643546    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-kube-api-access-zfc2f" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "kube-api-access-zfc2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:34.643733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.643670    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-podinfo" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:34.644302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.644280    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-token" pod "26412140-b8b8-428b-802f-4e1a66f8fd79" (UID: "26412140-b8b8-428b-802f-4e1a66f8fd79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:34.647830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.647703    4519 scope.go:122] "RemoveContainer" containerID="3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f"
Oct 30 20:25:34.665920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.665892    4519 scope.go:122] "RemoveContainer" containerID="bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b"
Oct 30 20:25:34.666223 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:34.666203    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b\": container with ID starting with bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b not found: ID does not exist" containerID="bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b"
Oct 30 20:25:34.666317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.666233    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b"} err="failed to get container status \"bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b\": rpc error: code = NotFound desc = could not find container \"bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b\": container with ID starting with bf02e8f9b9f2e350e9872a01c98ee46e460f5b0c9d93742e34efa838f232746b not found: ID does not exist"
Oct 30 20:25:34.666317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.666253    4519 scope.go:122] "RemoveContainer" containerID="3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f"
Oct 30 20:25:34.666545 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:34.666529    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f\": container with ID starting with 3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f not found: ID does not exist" containerID="3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f"
Oct 30 20:25:34.666628 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.666549    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f"} err="failed to get container status \"3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f\": rpc error: code = NotFound desc = could not find container \"3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f\": container with ID starting with 3c60b9b21e291d11be1891948730e5995ed7c1091bb7682ff56649aa7d32e62f not found: ID does not exist"
Oct 30 20:25:34.722907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.722876    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"]
Oct 30 20:25:34.732170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.732144    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"]
Oct 30 20:25:34.743783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.743756    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.743783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.743782    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.743999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.743791    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26412140-b8b8-428b-802f-4e1a66f8fd79-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.743999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.743814    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfc2f\" (UniqueName: \"kubernetes.io/projected/26412140-b8b8-428b-802f-4e1a66f8fd79-kube-api-access-zfc2f\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.992418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.992364    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"]
Oct 30 20:25:34.995183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:34.995155    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7eec8-openshift-default-5b7fdc9969-6phw4"]
Oct 30 20:25:35.615433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:35.615400    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" path="/var/lib/kubelet/pods/26412140-b8b8-428b-802f-4e1a66f8fd79/volumes"
Oct 30 20:25:42.454438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.454407    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 20:25:42.509897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.509864    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a524b77-1bd9-45cd-98d9-699ade653c7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a524b77-1bd9-45cd-98d9-699ade653c7c-plugins-config-volume\") pod \"2a524b77-1bd9-45cd-98d9-699ade653c7c\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") "
Oct 30 20:25:42.510080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.510060    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a524b77-1bd9-45cd-98d9-699ade653c7c-kube-api-access-mtf94\" (UniqueName: \"kubernetes.io/projected/2a524b77-1bd9-45cd-98d9-699ade653c7c-kube-api-access-mtf94\") pod \"2a524b77-1bd9-45cd-98d9-699ade653c7c\" (UID: \"2a524b77-1bd9-45cd-98d9-699ade653c7c\") "
Oct 30 20:25:42.510302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.510274    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a524b77-1bd9-45cd-98d9-699ade653c7c-plugins-config-volume" pod "2a524b77-1bd9-45cd-98d9-699ade653c7c" (UID: "2a524b77-1bd9-45cd-98d9-699ade653c7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:42.510633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.510612    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a524b77-1bd9-45cd-98d9-699ade653c7c-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:42.512324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.512295    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a524b77-1bd9-45cd-98d9-699ade653c7c-kube-api-access-mtf94" pod "2a524b77-1bd9-45cd-98d9-699ade653c7c" (UID: "2a524b77-1bd9-45cd-98d9-699ade653c7c"). InnerVolumeSpecName "kube-api-access-mtf94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:42.611677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.611641    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtf94\" (UniqueName: \"kubernetes.io/projected/2a524b77-1bd9-45cd-98d9-699ade653c7c-kube-api-access-mtf94\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:42.731114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.731015    4519 generic.go:356] "Generic (PLEG): container finished" podID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerID="05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774" exitCode=0
Oct 30 20:25:42.731114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.731066    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" event={"ID":"2a524b77-1bd9-45cd-98d9-699ade653c7c","Type":"ContainerDied","Data":"05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774"}
Oct 30 20:25:42.731114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.731071    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"
Oct 30 20:25:42.731114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.731105    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" event={"ID":"2a524b77-1bd9-45cd-98d9-699ade653c7c","Type":"ContainerDied","Data":"612eed5b785129ce710e1fa5dab2a36e00d49de8eef98e30e12714651d012508"}
Oct 30 20:25:42.731448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.731128    4519 scope.go:122] "RemoveContainer" containerID="05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774"
Oct 30 20:25:42.742414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.742395    4519 scope.go:122] "RemoveContainer" containerID="05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774"
Oct 30 20:25:42.742719 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:42.742698    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774\": container with ID starting with 05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774 not found: ID does not exist" containerID="05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774"
Oct 30 20:25:42.742817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.742726    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774"} err="failed to get container status \"05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774\": rpc error: code = NotFound desc = could not find container \"05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774\": container with ID starting with 05bac2d77801abda4acf985fa7274e56a9d1d0fd3244ac97efdb0d233cd4b774 not found: ID does not exist"
Oct 30 20:25:42.785538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.785506    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"]
Oct 30 20:25:42.789460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:42.789436    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx"]
Oct 30 20:25:43.441354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:43.441306    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a3426-5ffb7d8df9-nkftx" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded"
Oct 30 20:25:43.441577 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:43.441432    4519 logging.go:55] [core] [Channel #1433 SubChannel #1434]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.54:50052: operation was canceled"
Oct 30 20:25:43.613057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:43.613021    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" path="/var/lib/kubelet/pods/2a524b77-1bd9-45cd-98d9-699ade653c7c/volumes"
Oct 30 20:25:44.930361 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:44.930330    4519 scope.go:122] "RemoveContainer" containerID="bfca253ec0e8974d341a547333739ec6e0669e07b9003aecfc29224ba92f8b86"
Oct 30 20:25:49.745228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:49.745179    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" containerID="cri-o://b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b" gracePeriod=45
Oct 30 20:25:49.756040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:49.755982    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" containerID="cri-o://86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200" gracePeriod=45
Oct 30 20:25:50.171084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.171045    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 20:25:50.175819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.175782    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 20:25:50.263902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.263815    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tls-certs\") pod \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") "
Oct 30 20:25:50.263902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.263874    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-home\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-home\") pod \"a974082e-4a02-4a95-9a91-a2f97228a9f2\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") "
Oct 30 20:25:50.264148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.263950    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-home\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-home\") pod \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") "
Oct 30 20:25:50.264148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.263985    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-model-cache\") pod \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") "
Oct 30 20:25:50.264148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264025    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-dshm\") pod \"a974082e-4a02-4a95-9a91-a2f97228a9f2\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") "
Oct 30 20:25:50.264148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264058    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a974082e-4a02-4a95-9a91-a2f97228a9f2-kube-api-access-xh578\" (UniqueName: \"kubernetes.io/projected/a974082e-4a02-4a95-9a91-a2f97228a9f2-kube-api-access-xh578\") pod \"a974082e-4a02-4a95-9a91-a2f97228a9f2\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") "
Oct 30 20:25:50.264148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264114    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-tmp-dir\") pod \"a974082e-4a02-4a95-9a91-a2f97228a9f2\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") "
Oct 30 20:25:50.264431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264205    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-home" pod "a974082e-4a02-4a95-9a91-a2f97228a9f2" (UID: "a974082e-4a02-4a95-9a91-a2f97228a9f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.264431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264274    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a974082e-4a02-4a95-9a91-a2f97228a9f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a974082e-4a02-4a95-9a91-a2f97228a9f2-tls-certs\") pod \"a974082e-4a02-4a95-9a91-a2f97228a9f2\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") "
Oct 30 20:25:50.264431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264325    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-model-cache\") pod \"a974082e-4a02-4a95-9a91-a2f97228a9f2\" (UID: \"a974082e-4a02-4a95-9a91-a2f97228a9f2\") "
Oct 30 20:25:50.264431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264355    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-dshm\") pod \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") "
Oct 30 20:25:50.264431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264428    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tmp-dir\") pod \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") "
Oct 30 20:25:50.264881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264460    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-tmp-dir" pod "a974082e-4a02-4a95-9a91-a2f97228a9f2" (UID: "a974082e-4a02-4a95-9a91-a2f97228a9f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.264881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264460    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1f16b8-14c6-4dc5-83fd-901a42afade4-kube-api-access-zdcnh\" (UniqueName: \"kubernetes.io/projected/8a1f16b8-14c6-4dc5-83fd-901a42afade4-kube-api-access-zdcnh\") pod \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\" (UID: \"8a1f16b8-14c6-4dc5-83fd-901a42afade4\") "
Oct 30 20:25:50.264881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264493    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-model-cache" pod "8a1f16b8-14c6-4dc5-83fd-901a42afade4" (UID: "8a1f16b8-14c6-4dc5-83fd-901a42afade4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.264881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.264712    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-home" pod "8a1f16b8-14c6-4dc5-83fd-901a42afade4" (UID: "8a1f16b8-14c6-4dc5-83fd-901a42afade4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.265170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.265151    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.265255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.265174    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.265255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.265187    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.265255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.265201    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.265582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.265551    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-model-cache" pod "a974082e-4a02-4a95-9a91-a2f97228a9f2" (UID: "a974082e-4a02-4a95-9a91-a2f97228a9f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.266803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.266771    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tmp-dir" pod "8a1f16b8-14c6-4dc5-83fd-901a42afade4" (UID: "8a1f16b8-14c6-4dc5-83fd-901a42afade4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.267910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.267881    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a974082e-4a02-4a95-9a91-a2f97228a9f2-kube-api-access-xh578" pod "a974082e-4a02-4a95-9a91-a2f97228a9f2" (UID: "a974082e-4a02-4a95-9a91-a2f97228a9f2"). InnerVolumeSpecName "kube-api-access-xh578". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:50.267991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.267880    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tls-certs" pod "8a1f16b8-14c6-4dc5-83fd-901a42afade4" (UID: "8a1f16b8-14c6-4dc5-83fd-901a42afade4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:50.268027 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.267991    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-dshm" pod "a974082e-4a02-4a95-9a91-a2f97228a9f2" (UID: "a974082e-4a02-4a95-9a91-a2f97228a9f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.268222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.268195    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a974082e-4a02-4a95-9a91-a2f97228a9f2-tls-certs" pod "a974082e-4a02-4a95-9a91-a2f97228a9f2" (UID: "a974082e-4a02-4a95-9a91-a2f97228a9f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:50.269946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.269445    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1f16b8-14c6-4dc5-83fd-901a42afade4-kube-api-access-zdcnh" pod "8a1f16b8-14c6-4dc5-83fd-901a42afade4" (UID: "8a1f16b8-14c6-4dc5-83fd-901a42afade4"). InnerVolumeSpecName "kube-api-access-zdcnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:50.269946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.269798    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-dshm" pod "8a1f16b8-14c6-4dc5-83fd-901a42afade4" (UID: "8a1f16b8-14c6-4dc5-83fd-901a42afade4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:50.366693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366654    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.366693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366695    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh578\" (UniqueName: \"kubernetes.io/projected/a974082e-4a02-4a95-9a91-a2f97228a9f2-kube-api-access-xh578\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.366923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366708    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a974082e-4a02-4a95-9a91-a2f97228a9f2-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.366923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366725    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a974082e-4a02-4a95-9a91-a2f97228a9f2-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.366923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366736    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.366923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366746    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.366923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366756    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdcnh\" (UniqueName: \"kubernetes.io/projected/8a1f16b8-14c6-4dc5-83fd-901a42afade4-kube-api-access-zdcnh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.366923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.366768    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1f16b8-14c6-4dc5-83fd-901a42afade4-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:25:50.907007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.906967    4519 generic.go:356] "Generic (PLEG): container finished" podID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerID="86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200" exitCode=0
Oct 30 20:25:50.907007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.907010    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"
Oct 30 20:25:50.907617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.907050    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" event={"ID":"a974082e-4a02-4a95-9a91-a2f97228a9f2","Type":"ContainerDied","Data":"86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200"}
Oct 30 20:25:50.907617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.907097    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" event={"ID":"a974082e-4a02-4a95-9a91-a2f97228a9f2","Type":"ContainerDied","Data":"62a92535b3069663b27deb831f985b0afd0bbe33c423b9e6efdc0d78abf63c78"}
Oct 30 20:25:50.907617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.907132    4519 scope.go:122] "RemoveContainer" containerID="86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200"
Oct 30 20:25:50.909714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.909523    4519 generic.go:356] "Generic (PLEG): container finished" podID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerID="b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b" exitCode=0
Oct 30 20:25:50.909714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.909557    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" event={"ID":"8a1f16b8-14c6-4dc5-83fd-901a42afade4","Type":"ContainerDied","Data":"b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b"}
Oct 30 20:25:50.909714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.909585    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" event={"ID":"8a1f16b8-14c6-4dc5-83fd-901a42afade4","Type":"ContainerDied","Data":"a135e3549bb4101317c5d79e303757174e3d22d762df2d6abb098a3a7231c4d6"}
Oct 30 20:25:50.909714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.909662    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"
Oct 30 20:25:50.974545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:50.974513    4519 scope.go:122] "RemoveContainer" containerID="86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200"
Oct 30 20:25:51.014680 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:51.014636    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200\": container with ID starting with 86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200 not found: ID does not exist" containerID="86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200"
Oct 30 20:25:51.014926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.014896    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200"} err="failed to get container status \"86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200\": rpc error: code = NotFound desc = could not find container \"86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200\": container with ID starting with 86390b0fc47e2e4b2556e7ff0219ad70b894e6890074bd7a76745db9ba4fd200 not found: ID does not exist"
Oct 30 20:25:51.015066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.015053    4519 scope.go:122] "RemoveContainer" containerID="b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b"
Oct 30 20:25:51.075393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.075363    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"]
Oct 30 20:25:51.129407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.129380    4519 scope.go:122] "RemoveContainer" containerID="b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b"
Oct 30 20:25:51.130115 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:25:51.130088    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b\": container with ID starting with b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b not found: ID does not exist" containerID="b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b"
Oct 30 20:25:51.130276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.130250    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b"} err="failed to get container status \"b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b\": rpc error: code = NotFound desc = could not find container \"b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b\": container with ID starting with b3d07d51b80733674fe774f6f22f2873f389e8d7fa3c49e113aa8fddc406f48b not found: ID does not exist"
Oct 30 20:25:51.152502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.152442    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:25:51.153204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.152731    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:25:51.215696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.209536    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/premium-w5-6c4dd1-kserve-bcb6d975c-cwgm8"]
Oct 30 20:25:51.247489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.240844    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"]
Oct 30 20:25:51.259486 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.258508    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6c4dd1/simulator-w5-6c4dd1-kserve-866b58b7cb-cw22w"]
Oct 30 20:25:51.626491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.625082    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" path="/var/lib/kubelet/pods/8a1f16b8-14c6-4dc5-83fd-901a42afade4/volumes"
Oct 30 20:25:51.626491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:51.625923    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" path="/var/lib/kubelet/pods/a974082e-4a02-4a95-9a91-a2f97228a9f2/volumes"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908065    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"]
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908642    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerName="payload-pre-processing"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908658    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerName="payload-pre-processing"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908672    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908677    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39e72f2-f9c1-4d6a-b581-99d1a2555536" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908686    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908691    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908707    4519 cpu_manager.go:414] "containerMap: removing container" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908712    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908729    4519 cpu_manager.go:414] "containerMap: removing container" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908736    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908754    4519 cpu_manager.go:414] "containerMap: removing container" podUID="dd74ffd0-9048-4dac-96b2-d14f5ba3703a" containerName="revoke-keys"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908760    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd74ffd0-9048-4dac-96b2-d14f5ba3703a" containerName="revoke-keys"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908770    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908774    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908854    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a524b77-1bd9-45cd-98d9-699ade653c7c" containerName="payload-pre-processing"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908865    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a974082e-4a02-4a95-9a91-a2f97228a9f2" containerName="main"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908873    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908881    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="26412140-b8b8-428b-802f-4e1a66f8fd79" containerName="istio-proxy"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908888    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd74ffd0-9048-4dac-96b2-d14f5ba3703a" containerName="revoke-keys"
Oct 30 20:25:53.910198 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.908894    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f16b8-14c6-4dc5-83fd-901a42afade4" containerName="main"
Oct 30 20:25:53.914391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.914366    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:53.933044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.932733    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"]
Oct 30 20:25:53.939608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.939582    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"openshift-service-ca.crt\""
Oct 30 20:25:53.939812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.939787    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"rate-test-model-5314c2-kserve-self-signed-certs\""
Oct 30 20:25:53.942360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:53.942335    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"default-dockercfg-92gh8\""
Oct 30 20:25:54.029925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.027106    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-dshm\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.029925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.027171    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-tmp-dir\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.029925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.027266    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-model-cache\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.029925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.027350    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-home\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.029925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.027414    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6854c09-8a7c-4668-844d-7ce7ac766060-tls-certs\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.029925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.027550    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2gl\" (UniqueName: \"kubernetes.io/projected/c6854c09-8a7c-4668-844d-7ce7ac766060-kube-api-access-tm2gl\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.030820 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.030797    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"kube-root-ca.crt\""
Oct 30 20:25:54.129490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.128546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-dshm\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.129490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.128605    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-tmp-dir\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.129490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.128633    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-model-cache\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.129490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.128674    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-home\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.129490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.128730    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6854c09-8a7c-4668-844d-7ce7ac766060-tls-certs\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.129490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.128836    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2gl\" (UniqueName: \"kubernetes.io/projected/c6854c09-8a7c-4668-844d-7ce7ac766060-kube-api-access-tm2gl\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.133525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.131865    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-home\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.133525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.132125    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-tmp-dir\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.133525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.132231    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-model-cache\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.135216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.134411    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-dshm\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.135216 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.134890    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6854c09-8a7c-4668-844d-7ce7ac766060-tls-certs\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.158552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.154158    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2gl\" (UniqueName: \"kubernetes.io/projected/c6854c09-8a7c-4668-844d-7ce7ac766060-kube-api-access-tm2gl\") pod \"rate-test-model-5314c2-kserve-66bb55ff97-vflj8\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.234307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.233158    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:54.815533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:54.815497    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"]
Oct 30 20:25:54.817537 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:54.817505    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6854c09_8a7c_4668_844d_7ce7ac766060.slice/crio-2646808a89daf698012a6af7b2c2920ac4f83a43af59aa46493dd4995c9739a6 WatchSource:0}: Error finding container 2646808a89daf698012a6af7b2c2920ac4f83a43af59aa46493dd4995c9739a6: Status 404 returned error can't find the container with id 2646808a89daf698012a6af7b2c2920ac4f83a43af59aa46493dd4995c9739a6
Oct 30 20:25:55.041744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:55.041707    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8" event={"ID":"c6854c09-8a7c-4668-844d-7ce7ac766060","Type":"ContainerStarted","Data":"983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d"}
Oct 30 20:25:55.042118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:55.041753    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8" event={"ID":"c6854c09-8a7c-4668-844d-7ce7ac766060","Type":"ContainerStarted","Data":"2646808a89daf698012a6af7b2c2920ac4f83a43af59aa46493dd4995c9739a6"}
Oct 30 20:25:55.108306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:55.108167    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8" podStartSLOduration=2.108150547 podStartE2EDuration="2.108150547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:55.09836142 +0000 UTC m=+3079.599065147" watchObservedRunningTime="2026-10-30 20:25:55.108150547 +0000 UTC m=+3079.608854296"
Oct 30 20:25:55.235731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:55.235690    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:25:58.574136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.574096    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"]
Oct 30 20:25:58.579150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.579124    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.591961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.591226    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"]
Oct 30 20:25:58.596068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.596037    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.604251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.603130    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"]
Oct 30 20:25:58.604251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.604024    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:25:58.608994 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.608972    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:25:58.620054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.620022    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"]
Oct 30 20:25:58.631673 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.631643    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d9516f-c965-469c-a75b-346b0684801d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.631819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.631746    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhq2\" (UniqueName: \"kubernetes.io/projected/5764e96d-e897-40d5-a387-c8aec9042b43-kube-api-access-jfhq2\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.631819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.631789    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6pl\" (UniqueName: \"kubernetes.io/projected/71d9516f-c965-469c-a75b-346b0684801d-kube-api-access-zp6pl\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.631819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.631811    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d9516f-c965-469c-a75b-346b0684801d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.632013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.631862    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.632013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.631925    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.732947 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.732911    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d9516f-c965-469c-a75b-346b0684801d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.733118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.733049    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhq2\" (UniqueName: \"kubernetes.io/projected/5764e96d-e897-40d5-a387-c8aec9042b43-kube-api-access-jfhq2\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.733231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.733203    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6pl\" (UniqueName: \"kubernetes.io/projected/71d9516f-c965-469c-a75b-346b0684801d-kube-api-access-zp6pl\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.733301 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.733252    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d9516f-c965-469c-a75b-346b0684801d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.733388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.733371    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.733574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.733554    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.734455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.734430    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.734630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.734501    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d9516f-c965-469c-a75b-346b0684801d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.736496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.736434    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.736738 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.736668    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d9516f-c965-469c-a75b-346b0684801d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.760075 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.760053    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6pl\" (UniqueName: \"kubernetes.io/projected/71d9516f-c965-469c-a75b-346b0684801d-kube-api-access-zp6pl\") pod \"maas-api-revoke-sub-945865d9-sqf2d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.760411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.760394    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhq2\" (UniqueName: \"kubernetes.io/projected/5764e96d-e897-40d5-a387-c8aec9042b43-kube-api-access-jfhq2\") pod \"maas-api-revoke-sub-90c05905-9mchk\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:58.919476 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.919431    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:25:58.933878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:58.933845    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:25:59.267220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:59.267185    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"]
Oct 30 20:25:59.268425 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:59.268363    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d9516f_c965_469c_a75b_346b0684801d.slice/crio-9a7edefc1606aba321c8a61ee312235d03c20675bd09b3710bc8045f7b0a5a16 WatchSource:0}: Error finding container 9a7edefc1606aba321c8a61ee312235d03c20675bd09b3710bc8045f7b0a5a16: Status 404 returned error can't find the container with id 9a7edefc1606aba321c8a61ee312235d03c20675bd09b3710bc8045f7b0a5a16
Oct 30 20:25:59.292948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:59.292718    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"]
Oct 30 20:25:59.294334 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:25:59.294299    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5764e96d_e897_40d5_a387_c8aec9042b43.slice/crio-0c82e19274330bc8b4131d90c0cf959d863030163cdad1f2969bee2922cde999 WatchSource:0}: Error finding container 0c82e19274330bc8b4131d90c0cf959d863030163cdad1f2969bee2922cde999: Status 404 returned error can't find the container with id 0c82e19274330bc8b4131d90c0cf959d863030163cdad1f2969bee2922cde999
Oct 30 20:25:59.960758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:25:59.960718    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p_3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e/payload-processing/0.log"
Oct 30 20:26:00.177341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.174022    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d" event={"ID":"71d9516f-c965-469c-a75b-346b0684801d","Type":"ContainerStarted","Data":"70da87e188afd99be5c7fed7744169b00de9ca416c2a71c5310d4b690c43bda6"}
Oct 30 20:26:00.177341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.174124    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d" event={"ID":"71d9516f-c965-469c-a75b-346b0684801d","Type":"ContainerStarted","Data":"9a7edefc1606aba321c8a61ee312235d03c20675bd09b3710bc8045f7b0a5a16"}
Oct 30 20:26:00.177341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.176351    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk" event={"ID":"5764e96d-e897-40d5-a387-c8aec9042b43","Type":"ContainerStarted","Data":"c703d1173df035a8e8715780c7310d550cea4ddaa1740d142eef58d3b349dcf5"}
Oct 30 20:26:00.177341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.176565    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk" event={"ID":"5764e96d-e897-40d5-a387-c8aec9042b43","Type":"ContainerStarted","Data":"0c82e19274330bc8b4131d90c0cf959d863030163cdad1f2969bee2922cde999"}
Oct 30 20:26:00.218594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.218448    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d" podStartSLOduration=2.218431888 podStartE2EDuration="2.218431888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:00.215826493 +0000 UTC m=+3084.716530218" watchObservedRunningTime="2026-10-30 20:26:00.218431888 +0000 UTC m=+3084.719135612"
Oct 30 20:26:00.265503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.261503    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk" podStartSLOduration=2.261484618 podStartE2EDuration="2.261484618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:00.258768608 +0000 UTC m=+3084.759472333" watchObservedRunningTime="2026-10-30 20:26:00.261484618 +0000 UTC m=+3084.762188337"
Oct 30 20:26:00.901932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.901693    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:26:00.902149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:00.901982    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:26:01.190871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.190296    4519 generic.go:356] "Generic (PLEG): container finished" podID="71d9516f-c965-469c-a75b-346b0684801d" containerID="70da87e188afd99be5c7fed7744169b00de9ca416c2a71c5310d4b690c43bda6" exitCode=0
Oct 30 20:26:01.190871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.190443    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d" event={"ID":"71d9516f-c965-469c-a75b-346b0684801d","Type":"ContainerDied","Data":"70da87e188afd99be5c7fed7744169b00de9ca416c2a71c5310d4b690c43bda6"}
Oct 30 20:26:01.193298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.193262    4519 generic.go:356] "Generic (PLEG): container finished" podID="5764e96d-e897-40d5-a387-c8aec9042b43" containerID="c703d1173df035a8e8715780c7310d550cea4ddaa1740d142eef58d3b349dcf5" exitCode=0
Oct 30 20:26:01.193453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.193358    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk" event={"ID":"5764e96d-e897-40d5-a387-c8aec9042b43","Type":"ContainerDied","Data":"c703d1173df035a8e8715780c7310d550cea4ddaa1740d142eef58d3b349dcf5"}
Oct 30 20:26:01.621240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.621154    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"]
Oct 30 20:26:01.625287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.625266    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.645958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.645932    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"]
Oct 30 20:26:01.800542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.800499    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.802843 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.800711    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbwz\" (UniqueName: \"kubernetes.io/projected/860cdd40-231b-48b5-866e-cd2700dd183c-kube-api-access-qfbwz\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.802843 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.800884    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.902230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.902130    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbwz\" (UniqueName: \"kubernetes.io/projected/860cdd40-231b-48b5-866e-cd2700dd183c-kube-api-access-qfbwz\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.902422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.902323    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.902422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.902406    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.903555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.903529    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.905868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.905834    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.923203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.923168    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbwz\" (UniqueName: \"kubernetes.io/projected/860cdd40-231b-48b5-866e-cd2700dd183c-kube-api-access-qfbwz\") pod \"maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:01.978082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:01.978006    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:02.203356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.203314    4519 generic.go:356] "Generic (PLEG): container finished" podID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerID="0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee" exitCode=0
Oct 30 20:26:02.203830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.203400    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" event={"ID":"c6155870-2e23-4484-8c1d-1b34e19e2bca","Type":"ContainerDied","Data":"0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee"}
Oct 30 20:26:02.204163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.204141    4519 scope.go:122] "RemoveContainer" containerID="0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee"
Oct 30 20:26:02.206669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.206609    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:26:02.206801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.206712    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:26:02.206801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.206769    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:26:02.229719 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:26:02.229643    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:26:02.305159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.304307    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"]
Oct 30 20:26:02.462011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.461909    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:26:02.498852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.498817    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"]
Oct 30 20:26:02.499992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.499958    4519 cpu_manager.go:414] "containerMap: removing container" podUID="71d9516f-c965-469c-a75b-346b0684801d" containerName="revoke-keys"
Oct 30 20:26:02.499992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.499991    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d9516f-c965-469c-a75b-346b0684801d" containerName="revoke-keys"
Oct 30 20:26:02.500201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.500159    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d9516f-c965-469c-a75b-346b0684801d" containerName="revoke-keys"
Oct 30 20:26:02.504706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.504685    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.656554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.644895    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d9516f-c965-469c-a75b-346b0684801d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d9516f-c965-469c-a75b-346b0684801d-maas-api-xks-ca\") pod \"71d9516f-c965-469c-a75b-346b0684801d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") "
Oct 30 20:26:02.656554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.644996    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d9516f-c965-469c-a75b-346b0684801d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d9516f-c965-469c-a75b-346b0684801d-maas-api-openshift-ca\") pod \"71d9516f-c965-469c-a75b-346b0684801d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") "
Oct 30 20:26:02.656554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.645155    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d9516f-c965-469c-a75b-346b0684801d-kube-api-access-zp6pl\" (UniqueName: \"kubernetes.io/projected/71d9516f-c965-469c-a75b-346b0684801d-kube-api-access-zp6pl\") pod \"71d9516f-c965-469c-a75b-346b0684801d\" (UID: \"71d9516f-c965-469c-a75b-346b0684801d\") "
Oct 30 20:26:02.656554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.645390    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.656554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.645566    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrk64\" (UniqueName: \"kubernetes.io/projected/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-kube-api-access-wrk64\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.656554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.645623    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.656554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.653209    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d9516f-c965-469c-a75b-346b0684801d-maas-api-openshift-ca" pod "71d9516f-c965-469c-a75b-346b0684801d" (UID: "71d9516f-c965-469c-a75b-346b0684801d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:02.683763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.683217    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"]
Oct 30 20:26:02.684919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.684864    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d9516f-c965-469c-a75b-346b0684801d-kube-api-access-zp6pl" pod "71d9516f-c965-469c-a75b-346b0684801d" (UID: "71d9516f-c965-469c-a75b-346b0684801d"). InnerVolumeSpecName "kube-api-access-zp6pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:02.705866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.705828    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d9516f-c965-469c-a75b-346b0684801d-maas-api-xks-ca" pod "71d9516f-c965-469c-a75b-346b0684801d" (UID: "71d9516f-c965-469c-a75b-346b0684801d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:02.718822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.716815    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"]
Oct 30 20:26:02.742609 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.725655    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"]
Oct 30 20:26:02.742609 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.725775    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.742609 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.737904    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:26:02.783859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.783824    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-openshift-ca\") pod \"5764e96d-e897-40d5-a387-c8aec9042b43\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") "
Oct 30 20:26:02.784066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.784048    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5764e96d-e897-40d5-a387-c8aec9042b43-kube-api-access-jfhq2\" (UniqueName: \"kubernetes.io/projected/5764e96d-e897-40d5-a387-c8aec9042b43-kube-api-access-jfhq2\") pod \"5764e96d-e897-40d5-a387-c8aec9042b43\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") "
Oct 30 20:26:02.784152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.784132    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-xks-ca\") pod \"5764e96d-e897-40d5-a387-c8aec9042b43\" (UID: \"5764e96d-e897-40d5-a387-c8aec9042b43\") "
Oct 30 20:26:02.784535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.784509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.786902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.786873    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.787143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.787123    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrk64\" (UniqueName: \"kubernetes.io/projected/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-kube-api-access-wrk64\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.787297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.784693    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-openshift-ca" pod "5764e96d-e897-40d5-a387-c8aec9042b43" (UID: "5764e96d-e897-40d5-a387-c8aec9042b43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:02.787651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.787368    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jsc\" (UniqueName: \"kubernetes.io/projected/6468a513-718f-4a96-8f4a-8fc872a03026-kube-api-access-w9jsc\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.787854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.787828    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.787949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.787933    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.788246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.788226    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d9516f-c965-469c-a75b-346b0684801d-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:02.790365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.789656    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.790530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.790388    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d9516f-c965-469c-a75b-346b0684801d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:02.790651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.790633    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:02.790762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.790749    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6pl\" (UniqueName: \"kubernetes.io/projected/71d9516f-c965-469c-a75b-346b0684801d-kube-api-access-zp6pl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:02.793153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.793130    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.793608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.793536    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-xks-ca" pod "5764e96d-e897-40d5-a387-c8aec9042b43" (UID: "5764e96d-e897-40d5-a387-c8aec9042b43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:02.794135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.794111    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5764e96d-e897-40d5-a387-c8aec9042b43-kube-api-access-jfhq2" pod "5764e96d-e897-40d5-a387-c8aec9042b43" (UID: "5764e96d-e897-40d5-a387-c8aec9042b43"). InnerVolumeSpecName "kube-api-access-jfhq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:02.807906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.807886    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrk64\" (UniqueName: \"kubernetes.io/projected/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-kube-api-access-wrk64\") pod \"maas-api-revoke-sub-cb56acd5-z7ctg\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:02.894984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.893207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.894984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.893284    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jsc\" (UniqueName: \"kubernetes.io/projected/6468a513-718f-4a96-8f4a-8fc872a03026-kube-api-access-w9jsc\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.894984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.893371    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.894984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.894079    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhq2\" (UniqueName: \"kubernetes.io/projected/5764e96d-e897-40d5-a387-c8aec9042b43-kube-api-access-jfhq2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:02.894984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.894113    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5764e96d-e897-40d5-a387-c8aec9042b43-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:02.894984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.894555    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.902264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.902184    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:02.914410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:02.914378    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jsc\" (UniqueName: \"kubernetes.io/projected/6468a513-718f-4a96-8f4a-8fc872a03026-kube-api-access-w9jsc\") pod \"maas-api-revoke-sub-0d3b1543-98b2t\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:03.041443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.041327    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:03.086311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.086270    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:03.302024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.300759    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6" event={"ID":"860cdd40-231b-48b5-866e-cd2700dd183c","Type":"ContainerStarted","Data":"38fe93ddc8632d04f19b9c6ab54ccc0d6e67d0fb50de28ff2cce0263357082d5"}
Oct 30 20:26:03.302024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.300889    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6" event={"ID":"860cdd40-231b-48b5-866e-cd2700dd183c","Type":"ContainerStarted","Data":"50b87121b8d60d287c3ae3e93ad842e8ae7a0ee4e2390415401bcf3855ea1825"}
Oct 30 20:26:03.311990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.311953    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" event={"ID":"c6155870-2e23-4484-8c1d-1b34e19e2bca","Type":"ContainerStarted","Data":"e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845"}
Oct 30 20:26:03.322809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.322410    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"
Oct 30 20:26:03.322809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.322410    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d" event={"ID":"71d9516f-c965-469c-a75b-346b0684801d","Type":"ContainerDied","Data":"9a7edefc1606aba321c8a61ee312235d03c20675bd09b3710bc8045f7b0a5a16"}
Oct 30 20:26:03.322809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.322645    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a7edefc1606aba321c8a61ee312235d03c20675bd09b3710bc8045f7b0a5a16"
Oct 30 20:26:03.324901 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.324876    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk" event={"ID":"5764e96d-e897-40d5-a387-c8aec9042b43","Type":"ContainerDied","Data":"0c82e19274330bc8b4131d90c0cf959d863030163cdad1f2969bee2922cde999"}
Oct 30 20:26:03.325102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.325087    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c82e19274330bc8b4131d90c0cf959d863030163cdad1f2969bee2922cde999"
Oct 30 20:26:03.325848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.325707    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"
Oct 30 20:26:03.327747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.327718    4519 generic.go:356] "Generic (PLEG): container finished" podID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerID="1feaf89c42b04dbba2bd1d28a3ecadc86642385d7dc8384c3cd6258ff10c601d" exitCode=0
Oct 30 20:26:03.327921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.327905    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" event={"ID":"4b38681b-3ca5-4c6b-8e5a-ff36c5569665","Type":"ContainerDied","Data":"1feaf89c42b04dbba2bd1d28a3ecadc86642385d7dc8384c3cd6258ff10c601d"}
Oct 30 20:26:03.328426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.328408    4519 scope.go:122] "RemoveContainer" containerID="1feaf89c42b04dbba2bd1d28a3ecadc86642385d7dc8384c3cd6258ff10c601d"
Oct 30 20:26:03.331912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.331115    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:26:03.331912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.331219    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:26:03.331912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.331272    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:26:03.401747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.401696    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6" podStartSLOduration=2.401676873 podStartE2EDuration="2.401676873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:03.395707124 +0000 UTC m=+3087.896410848" watchObservedRunningTime="2026-10-30 20:26:03.401676873 +0000 UTC m=+3087.902380591"
Oct 30 20:26:03.508839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.508804    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"]
Oct 30 20:26:03.596607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:03.596515    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"]
Oct 30 20:26:03.617011 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:03.616970    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cba18c_e6d5_428e_a302_d6a90e6e87d5.slice/crio-0e5b45c509839b5ec1c50dc289185835b1812e561c290d4c5c0fb33677938cfc WatchSource:0}: Error finding container 0e5b45c509839b5ec1c50dc289185835b1812e561c290d4c5c0fb33677938cfc: Status 404 returned error can't find the container with id 0e5b45c509839b5ec1c50dc289185835b1812e561c290d4c5c0fb33677938cfc
Oct 30 20:26:04.230455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.230419    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:26:04.230673 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.230645    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 20:26:04.230736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.230707    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 20:26:04.233888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.233866    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:26:04.246153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.246132    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:26:04.338418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.338379    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t" event={"ID":"6468a513-718f-4a96-8f4a-8fc872a03026","Type":"ContainerStarted","Data":"f2e435b025bafbc9508bedc0573f7ac76af1bbd8e334cccad5932f4e23cc3aba"}
Oct 30 20:26:04.338954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.338442    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t" event={"ID":"6468a513-718f-4a96-8f4a-8fc872a03026","Type":"ContainerStarted","Data":"b5753d7315b7ee6b4a89c629522af93d676fc773c929bed6292dee473d02e601"}
Oct 30 20:26:04.340645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.340612    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" event={"ID":"4b38681b-3ca5-4c6b-8e5a-ff36c5569665","Type":"ContainerStarted","Data":"e8ee6ae805c71bdfc28d3bc10472db1c88fba43900538c8401e8e37fb66072ca"}
Oct 30 20:26:04.342307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.342245    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:26:04.342503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.342480    4519 generic.go:356] "Generic (PLEG): container finished" podID="860cdd40-231b-48b5-866e-cd2700dd183c" containerID="38fe93ddc8632d04f19b9c6ab54ccc0d6e67d0fb50de28ff2cce0263357082d5" exitCode=0
Oct 30 20:26:04.342587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.342561    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6" event={"ID":"860cdd40-231b-48b5-866e-cd2700dd183c","Type":"ContainerDied","Data":"38fe93ddc8632d04f19b9c6ab54ccc0d6e67d0fb50de28ff2cce0263357082d5"}
Oct 30 20:26:04.343976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.343949    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 20:26:04.344114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.344003    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 20:26:04.344511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.344459    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg" event={"ID":"f1cba18c-e6d5-428e-a302-d6a90e6e87d5","Type":"ContainerStarted","Data":"6f23f906d073cb743b00da8ec176a7dac8758cf46581beb90c5b1861aae7a5eb"}
Oct 30 20:26:04.344608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.344529    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg" event={"ID":"f1cba18c-e6d5-428e-a302-d6a90e6e87d5","Type":"ContainerStarted","Data":"0e5b45c509839b5ec1c50dc289185835b1812e561c290d4c5c0fb33677938cfc"}
Oct 30 20:26:04.390802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.390772    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:26:04.412068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.412020    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t" podStartSLOduration=2.412007214 podStartE2EDuration="2.412007214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:04.411713455 +0000 UTC m=+3088.912417178" watchObservedRunningTime="2026-10-30 20:26:04.412007214 +0000 UTC m=+3088.912710937"
Oct 30 20:26:04.510777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:04.510678    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg" podStartSLOduration=2.510664067 podStartE2EDuration="2.510664067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:04.509591656 +0000 UTC m=+3089.010295381" watchObservedRunningTime="2026-10-30 20:26:04.510664067 +0000 UTC m=+3089.011367790"
Oct 30 20:26:05.229952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.229912    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 20:26:05.230173 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.229977    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 20:26:05.343090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.343047    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 20:26:05.343528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.343111    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 20:26:05.382493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.379401    4519 generic.go:356] "Generic (PLEG): container finished" podID="f1cba18c-e6d5-428e-a302-d6a90e6e87d5" containerID="6f23f906d073cb743b00da8ec176a7dac8758cf46581beb90c5b1861aae7a5eb" exitCode=0
Oct 30 20:26:05.387445 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.383682    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg" event={"ID":"f1cba18c-e6d5-428e-a302-d6a90e6e87d5","Type":"ContainerDied","Data":"6f23f906d073cb743b00da8ec176a7dac8758cf46581beb90c5b1861aae7a5eb"}
Oct 30 20:26:05.387445 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.386174    4519 generic.go:356] "Generic (PLEG): container finished" podID="6468a513-718f-4a96-8f4a-8fc872a03026" containerID="f2e435b025bafbc9508bedc0573f7ac76af1bbd8e334cccad5932f4e23cc3aba" exitCode=0
Oct 30 20:26:05.387445 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.386574    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t" event={"ID":"6468a513-718f-4a96-8f4a-8fc872a03026","Type":"ContainerDied","Data":"f2e435b025bafbc9508bedc0573f7ac76af1bbd8e334cccad5932f4e23cc3aba"}
Oct 30 20:26:05.821545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.821433    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:05.844490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.844449    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-xks-ca\") pod \"860cdd40-231b-48b5-866e-cd2700dd183c\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") "
Oct 30 20:26:05.844649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.844525    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860cdd40-231b-48b5-866e-cd2700dd183c-kube-api-access-qfbwz\" (UniqueName: \"kubernetes.io/projected/860cdd40-231b-48b5-866e-cd2700dd183c-kube-api-access-qfbwz\") pod \"860cdd40-231b-48b5-866e-cd2700dd183c\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") "
Oct 30 20:26:05.844738 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.844669    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-openshift-ca\") pod \"860cdd40-231b-48b5-866e-cd2700dd183c\" (UID: \"860cdd40-231b-48b5-866e-cd2700dd183c\") "
Oct 30 20:26:05.845383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.845358    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-openshift-ca" pod "860cdd40-231b-48b5-866e-cd2700dd183c" (UID: "860cdd40-231b-48b5-866e-cd2700dd183c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:05.865869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.861635    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860cdd40-231b-48b5-866e-cd2700dd183c-kube-api-access-qfbwz" pod "860cdd40-231b-48b5-866e-cd2700dd183c" (UID: "860cdd40-231b-48b5-866e-cd2700dd183c"). InnerVolumeSpecName "kube-api-access-qfbwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:05.865869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.861721    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-xks-ca" pod "860cdd40-231b-48b5-866e-cd2700dd183c" (UID: "860cdd40-231b-48b5-866e-cd2700dd183c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:05.946402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.946361    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:05.946402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.946394    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860cdd40-231b-48b5-866e-cd2700dd183c-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:05.946402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:05.946407    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbwz\" (UniqueName: \"kubernetes.io/projected/860cdd40-231b-48b5-866e-cd2700dd183c-kube-api-access-qfbwz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:06.236489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.235685    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 20:26:06.236489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.235769    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 20:26:06.342806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.342770    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 20:26:06.343000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.342824    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 20:26:06.417516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.417443    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"
Oct 30 20:26:06.418040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.417633    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6" event={"ID":"860cdd40-231b-48b5-866e-cd2700dd183c","Type":"ContainerDied","Data":"50b87121b8d60d287c3ae3e93ad842e8ae7a0ee4e2390415401bcf3855ea1825"}
Oct 30 20:26:06.418040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.417704    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b87121b8d60d287c3ae3e93ad842e8ae7a0ee4e2390415401bcf3855ea1825"
Oct 30 20:26:06.908514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.898820    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:06.920523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:06.919723    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:07.022932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.022891    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6468a513-718f-4a96-8f4a-8fc872a03026-kube-api-access-w9jsc\" (UniqueName: \"kubernetes.io/projected/6468a513-718f-4a96-8f4a-8fc872a03026-kube-api-access-w9jsc\") pod \"6468a513-718f-4a96-8f4a-8fc872a03026\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") "
Oct 30 20:26:07.023145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.022958    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-kube-api-access-wrk64\" (UniqueName: \"kubernetes.io/projected/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-kube-api-access-wrk64\") pod \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") "
Oct 30 20:26:07.023145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.023023    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-openshift-ca\") pod \"6468a513-718f-4a96-8f4a-8fc872a03026\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") "
Oct 30 20:26:07.023145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.023094    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-xks-ca\") pod \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") "
Oct 30 20:26:07.023329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.023244    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-openshift-ca\") pod \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\" (UID: \"f1cba18c-e6d5-428e-a302-d6a90e6e87d5\") "
Oct 30 20:26:07.023329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.023318    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-xks-ca\") pod \"6468a513-718f-4a96-8f4a-8fc872a03026\" (UID: \"6468a513-718f-4a96-8f4a-8fc872a03026\") "
Oct 30 20:26:07.024753 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.023997    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-openshift-ca" pod "6468a513-718f-4a96-8f4a-8fc872a03026" (UID: "6468a513-718f-4a96-8f4a-8fc872a03026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:07.024753 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.024345    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-openshift-ca" pod "f1cba18c-e6d5-428e-a302-d6a90e6e87d5" (UID: "f1cba18c-e6d5-428e-a302-d6a90e6e87d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:07.027416 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.027387    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-xks-ca" pod "f1cba18c-e6d5-428e-a302-d6a90e6e87d5" (UID: "f1cba18c-e6d5-428e-a302-d6a90e6e87d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:07.027572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.027551    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-kube-api-access-wrk64" pod "f1cba18c-e6d5-428e-a302-d6a90e6e87d5" (UID: "f1cba18c-e6d5-428e-a302-d6a90e6e87d5"). InnerVolumeSpecName "kube-api-access-wrk64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:07.028141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.028106    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6468a513-718f-4a96-8f4a-8fc872a03026-kube-api-access-w9jsc" pod "6468a513-718f-4a96-8f4a-8fc872a03026" (UID: "6468a513-718f-4a96-8f4a-8fc872a03026"). InnerVolumeSpecName "kube-api-access-w9jsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:07.028633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.028607    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-xks-ca" pod "6468a513-718f-4a96-8f4a-8fc872a03026" (UID: "6468a513-718f-4a96-8f4a-8fc872a03026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:07.128145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.125977    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:07.128145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.126011    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:07.128145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.126027    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jsc\" (UniqueName: \"kubernetes.io/projected/6468a513-718f-4a96-8f4a-8fc872a03026-kube-api-access-w9jsc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:07.128145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.126040    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrk64\" (UniqueName: \"kubernetes.io/projected/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-kube-api-access-wrk64\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:07.128145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.126054    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6468a513-718f-4a96-8f4a-8fc872a03026-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:07.128145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.126069    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cba18c-e6d5-428e-a302-d6a90e6e87d5-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:07.230656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.230622    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 20:26:07.230881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.230683    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 20:26:07.342836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.342800    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 20:26:07.343008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.342862    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 20:26:07.441525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.438831    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t" event={"ID":"6468a513-718f-4a96-8f4a-8fc872a03026","Type":"ContainerDied","Data":"b5753d7315b7ee6b4a89c629522af93d676fc773c929bed6292dee473d02e601"}
Oct 30 20:26:07.441525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.438897    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5753d7315b7ee6b4a89c629522af93d676fc773c929bed6292dee473d02e601"
Oct 30 20:26:07.441525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.438975    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"
Oct 30 20:26:07.459601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.454877    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg" event={"ID":"f1cba18c-e6d5-428e-a302-d6a90e6e87d5","Type":"ContainerDied","Data":"0e5b45c509839b5ec1c50dc289185835b1812e561c290d4c5c0fb33677938cfc"}
Oct 30 20:26:07.479519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.474606    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e5b45c509839b5ec1c50dc289185835b1812e561c290d4c5c0fb33677938cfc"
Oct 30 20:26:07.479519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:07.459872    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"
Oct 30 20:26:08.230488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:08.229941    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 20:26:08.230488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:08.230058    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 20:26:08.342536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:08.342498    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 20:26:08.342702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:08.342564    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 20:26:08.511551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:08.508733    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:26:09.235581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:09.235549    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:26:09.236066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:09.236049    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:26:09.236784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:09.236758    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:26:09.343451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:09.343416    4519 patch_prober.go:29] interesting pod/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 20:26:09.343661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:09.343508    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 20:26:10.600953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:10.600911    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:26:10.603958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:10.603930    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.534513    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"]
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535313    4519 cpu_manager.go:414] "containerMap: removing container" podUID="860cdd40-231b-48b5-866e-cd2700dd183c" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535339    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860cdd40-231b-48b5-866e-cd2700dd183c" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535377    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5764e96d-e897-40d5-a387-c8aec9042b43" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535385    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5764e96d-e897-40d5-a387-c8aec9042b43" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535406    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6468a513-718f-4a96-8f4a-8fc872a03026" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535416    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6468a513-718f-4a96-8f4a-8fc872a03026" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535458    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f1cba18c-e6d5-428e-a302-d6a90e6e87d5" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535485    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cba18c-e6d5-428e-a302-d6a90e6e87d5" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535629    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5764e96d-e897-40d5-a387-c8aec9042b43" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535650    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6468a513-718f-4a96-8f4a-8fc872a03026" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535666    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cba18c-e6d5-428e-a302-d6a90e6e87d5" containerName="revoke-keys"
Oct 30 20:26:12.546494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.535676    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="860cdd40-231b-48b5-866e-cd2700dd183c" containerName="revoke-keys"
Oct 30 20:26:12.610490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.592455    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:12.610490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.601513    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"]
Oct 30 20:26:12.619936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.611079    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f35cc1\""
Oct 30 20:26:12.669493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.668063    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f35cc1-dockercfg-9nbgg\""
Oct 30 20:26:12.688301 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.688268    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"]
Oct 30 20:26:12.689908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.689874    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptxh\" (UniqueName: \"kubernetes.io/projected/5e5fdd18-17f4-4aeb-9ce3-42e134996662-kube-api-access-tptxh\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:12.690043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.690021    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5fdd18-17f4-4aeb-9ce3-42e134996662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:12.693130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.693107    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:12.721576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.721539    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"]
Oct 30 20:26:12.791768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.791723    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5fdd18-17f4-4aeb-9ce3-42e134996662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:12.791991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.791926    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:12.791991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.791980    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtxm\" (UniqueName: \"kubernetes.io/projected/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-kube-api-access-zhtxm\") pod \"payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:12.792130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.792029    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tptxh\" (UniqueName: \"kubernetes.io/projected/5e5fdd18-17f4-4aeb-9ce3-42e134996662-kube-api-access-tptxh\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:12.793358 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.793325    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5fdd18-17f4-4aeb-9ce3-42e134996662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:12.812437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.812349    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptxh\" (UniqueName: \"kubernetes.io/projected/5e5fdd18-17f4-4aeb-9ce3-42e134996662-kube-api-access-tptxh\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:12.893810 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.893762    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:12.893999 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.893847    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtxm\" (UniqueName: \"kubernetes.io/projected/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-kube-api-access-zhtxm\") pod \"payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:12.894876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.894843    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:12.913341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.913313    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtxm\" (UniqueName: \"kubernetes.io/projected/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-kube-api-access-zhtxm\") pod \"payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:12.969993 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:12.969955    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:13.008221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.008187    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:13.295434 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:13.294872    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5fdd18_17f4_4aeb_9ce3_42e134996662.slice/crio-f8cf5fe746c41965ce84a23abedd623f838dbc46981fdc02b9bcda49eafa532a WatchSource:0}: Error finding container f8cf5fe746c41965ce84a23abedd623f838dbc46981fdc02b9bcda49eafa532a: Status 404 returned error can't find the container with id f8cf5fe746c41965ce84a23abedd623f838dbc46981fdc02b9bcda49eafa532a
Oct 30 20:26:13.299829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.299798    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"]
Oct 30 20:26:13.364969 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:13.364911    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbdb14d_001a_4e7b_a922_2ecbe27f3f30.slice/crio-d53f69ffee8dd668b21a1e95f518adc51095aa4b7f3996765e5ab9f6a0d17f64 WatchSource:0}: Error finding container d53f69ffee8dd668b21a1e95f518adc51095aa4b7f3996765e5ab9f6a0d17f64: Status 404 returned error can't find the container with id d53f69ffee8dd668b21a1e95f518adc51095aa4b7f3996765e5ab9f6a0d17f64
Oct 30 20:26:13.377712 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.377682    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"]
Oct 30 20:26:13.603929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.603832    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg" event={"ID":"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30","Type":"ContainerStarted","Data":"ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3"}
Oct 30 20:26:13.603929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.603901    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg" event={"ID":"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30","Type":"ContainerStarted","Data":"d53f69ffee8dd668b21a1e95f518adc51095aa4b7f3996765e5ab9f6a0d17f64"}
Oct 30 20:26:13.604493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.603969    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:13.613182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.613153    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:13.613314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.613197    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq" event={"ID":"5e5fdd18-17f4-4aeb-9ce3-42e134996662","Type":"ContainerStarted","Data":"87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6"}
Oct 30 20:26:13.613314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.613220    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq" event={"ID":"5e5fdd18-17f4-4aeb-9ce3-42e134996662","Type":"ContainerStarted","Data":"f8cf5fe746c41965ce84a23abedd623f838dbc46981fdc02b9bcda49eafa532a"}
Oct 30 20:26:13.657252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:13.657194    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg" podStartSLOduration=1.657175426 podStartE2EDuration="1.657175426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:13.64917974 +0000 UTC m=+3098.149883465" watchObservedRunningTime="2026-10-30 20:26:13.657175426 +0000 UTC m=+3098.157879151"
Oct 30 20:26:15.136030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.135972    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq" podStartSLOduration=3.135953257 podStartE2EDuration="3.135953257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:13.694988083 +0000 UTC m=+3098.195691811" watchObservedRunningTime="2026-10-30 20:26:15.135953257 +0000 UTC m=+3099.636656981"
Oct 30 20:26:15.140526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.140497    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"]
Oct 30 20:26:15.145084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.145059    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.173487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.169337    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"]
Oct 30 20:26:15.173487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.173321    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"openshift-service-ca.crt\""
Oct 30 20:26:15.173487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.173373    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"default-dockercfg-72phh\""
Oct 30 20:26:15.173487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.173437    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"rate-test-model-77ac5a-kserve-self-signed-certs\""
Oct 30 20:26:15.176181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.176130    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"kube-root-ca.crt\""
Oct 30 20:26:15.230858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.230819    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tls-certs\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.231004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.230887    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tjd6\" (UniqueName: \"kubernetes.io/projected/efe0516c-11f3-4c15-bc16-a60322bdbcbe-kube-api-access-7tjd6\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.231004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.230961    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tmp-dir\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.231146 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.231131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-model-cache\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.231206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.231195    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-home\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.231245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.231233    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-dshm\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.332310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.332263    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tjd6\" (UniqueName: \"kubernetes.io/projected/efe0516c-11f3-4c15-bc16-a60322bdbcbe-kube-api-access-7tjd6\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.332656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.332616    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tmp-dir\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.332867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.332837    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-model-cache\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.332939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.332928    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-home\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.333012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.332953    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-dshm\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.333067 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.333011    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tls-certs\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.334019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.333989    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-home\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.334220 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.334195    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tmp-dir\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.334332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.334298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-model-cache\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.336342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.336311    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-dshm\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.336559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.336533    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tls-certs\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.353939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.353882    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tjd6\" (UniqueName: \"kubernetes.io/projected/efe0516c-11f3-4c15-bc16-a60322bdbcbe-kube-api-access-7tjd6\") pod \"rate-test-model-77ac5a-kserve-77dbc55885-2zvq8\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.467025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.466731    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:15.813527 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:15.813485    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefe0516c_11f3_4c15_bc16_a60322bdbcbe.slice/crio-a741e69c3c49a650b7ee93fb56edd9bf10f107ae211d85cfe3d9e8af5efb2085 WatchSource:0}: Error finding container a741e69c3c49a650b7ee93fb56edd9bf10f107ae211d85cfe3d9e8af5efb2085: Status 404 returned error can't find the container with id a741e69c3c49a650b7ee93fb56edd9bf10f107ae211d85cfe3d9e8af5efb2085
Oct 30 20:26:15.815316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:15.815292    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"]
Oct 30 20:26:16.641848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:16.641806    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8" event={"ID":"efe0516c-11f3-4c15-bc16-a60322bdbcbe","Type":"ContainerStarted","Data":"e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af"}
Oct 30 20:26:16.642261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:16.641869    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8" event={"ID":"efe0516c-11f3-4c15-bc16-a60322bdbcbe","Type":"ContainerStarted","Data":"a741e69c3c49a650b7ee93fb56edd9bf10f107ae211d85cfe3d9e8af5efb2085"}
Oct 30 20:26:16.685479 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:16.685423    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8" podStartSLOduration=1.685408282 podStartE2EDuration="1.685408282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:16.681862205 +0000 UTC m=+3101.182565930" watchObservedRunningTime="2026-10-30 20:26:16.685408282 +0000 UTC m=+3101.186111997"
Oct 30 20:26:17.468093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:17.468063    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:22.149565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.149488    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"]
Oct 30 20:26:22.154237 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.154216    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.176049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.176014    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"]
Oct 30 20:26:22.194437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.194405    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:26:22.194677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.194546    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:26:22.241553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.241520    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.241714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.241690    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.241816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.241798    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kx4\" (UniqueName: \"kubernetes.io/projected/1db76303-5059-4795-8a31-6f05e25aea64-kube-api-access-v5kx4\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.343012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.342970    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kx4\" (UniqueName: \"kubernetes.io/projected/1db76303-5059-4795-8a31-6f05e25aea64-kube-api-access-v5kx4\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.343169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.343121    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.343263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.343245    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.343853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.343829    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.345749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.345729    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.364159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.364126    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kx4\" (UniqueName: \"kubernetes.io/projected/1db76303-5059-4795-8a31-6f05e25aea64-kube-api-access-v5kx4\") pod \"maas-api-revoke-sub-c9cad9e6-54xxh\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.469996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.469961    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:22.782076 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:22.782039    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db76303_5059_4795_8a31_6f05e25aea64.slice/crio-f65b445a5ae00fee807305f6fb00f172c7d6a1130634c102e74bdbafb758807b WatchSource:0}: Error finding container f65b445a5ae00fee807305f6fb00f172c7d6a1130634c102e74bdbafb758807b: Status 404 returned error can't find the container with id f65b445a5ae00fee807305f6fb00f172c7d6a1130634c102e74bdbafb758807b
Oct 30 20:26:22.783661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:22.783628    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"]
Oct 30 20:26:23.731537 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:23.730531    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh" event={"ID":"1db76303-5059-4795-8a31-6f05e25aea64","Type":"ContainerStarted","Data":"d08a5b186cb0e67fec19d748b7e95a3c1eaaaae90671eab535b9a744be3f8ada"}
Oct 30 20:26:23.731537 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:23.730603    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh" event={"ID":"1db76303-5059-4795-8a31-6f05e25aea64","Type":"ContainerStarted","Data":"f65b445a5ae00fee807305f6fb00f172c7d6a1130634c102e74bdbafb758807b"}
Oct 30 20:26:23.799998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:23.799924    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh" podStartSLOduration=1.799903591 podStartE2EDuration="1.799903591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:23.796748531 +0000 UTC m=+3108.297452269" watchObservedRunningTime="2026-10-30 20:26:23.799903591 +0000 UTC m=+3108.300607314"
Oct 30 20:26:24.619748 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:24.619714    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:26:24.742009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:24.741964    4519 generic.go:356] "Generic (PLEG): container finished" podID="1db76303-5059-4795-8a31-6f05e25aea64" containerID="d08a5b186cb0e67fec19d748b7e95a3c1eaaaae90671eab535b9a744be3f8ada" exitCode=0
Oct 30 20:26:24.742447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:24.742027    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh" event={"ID":"1db76303-5059-4795-8a31-6f05e25aea64","Type":"ContainerDied","Data":"d08a5b186cb0e67fec19d748b7e95a3c1eaaaae90671eab535b9a744be3f8ada"}
Oct 30 20:26:25.467828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:25.467792    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:25.481389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:25.481361    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:25.633577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:25.633545    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:26:25.774057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:25.773608    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:26:25.968861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:25.968821    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:26.113658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.113581    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db76303-5059-4795-8a31-6f05e25aea64-kube-api-access-v5kx4\" (UniqueName: \"kubernetes.io/projected/1db76303-5059-4795-8a31-6f05e25aea64-kube-api-access-v5kx4\") pod \"1db76303-5059-4795-8a31-6f05e25aea64\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") "
Oct 30 20:26:26.113895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.113873    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-openshift-ca\") pod \"1db76303-5059-4795-8a31-6f05e25aea64\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") "
Oct 30 20:26:26.114005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.113982    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-xks-ca\") pod \"1db76303-5059-4795-8a31-6f05e25aea64\" (UID: \"1db76303-5059-4795-8a31-6f05e25aea64\") "
Oct 30 20:26:26.114284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.114257    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-openshift-ca" pod "1db76303-5059-4795-8a31-6f05e25aea64" (UID: "1db76303-5059-4795-8a31-6f05e25aea64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:26.114572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.114552    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:26.116156 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.116128    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-xks-ca" pod "1db76303-5059-4795-8a31-6f05e25aea64" (UID: "1db76303-5059-4795-8a31-6f05e25aea64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:26.116279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.116222    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db76303-5059-4795-8a31-6f05e25aea64-kube-api-access-v5kx4" pod "1db76303-5059-4795-8a31-6f05e25aea64" (UID: "1db76303-5059-4795-8a31-6f05e25aea64"). InnerVolumeSpecName "kube-api-access-v5kx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:26.216141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.216101    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db76303-5059-4795-8a31-6f05e25aea64-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:26.216141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.216137    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5kx4\" (UniqueName: \"kubernetes.io/projected/1db76303-5059-4795-8a31-6f05e25aea64-kube-api-access-v5kx4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:26.767362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.767328    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"
Oct 30 20:26:26.767362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.767353    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh" event={"ID":"1db76303-5059-4795-8a31-6f05e25aea64","Type":"ContainerDied","Data":"f65b445a5ae00fee807305f6fb00f172c7d6a1130634c102e74bdbafb758807b"}
Oct 30 20:26:26.767674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:26.767413    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f65b445a5ae00fee807305f6fb00f172c7d6a1130634c102e74bdbafb758807b"
Oct 30 20:26:30.122292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.122245    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg_ddbdb14d-001a-4e7b-a922-2ecbe27f3f30/payload-processing/0.log"
Oct 30 20:26:30.930069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.930026    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"]
Oct 30 20:26:30.930992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.930960    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1db76303-5059-4795-8a31-6f05e25aea64" containerName="revoke-keys"
Oct 30 20:26:30.931127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.930993    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db76303-5059-4795-8a31-6f05e25aea64" containerName="revoke-keys"
Oct 30 20:26:30.931169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.931132    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db76303-5059-4795-8a31-6f05e25aea64" containerName="revoke-keys"
Oct 30 20:26:30.935227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.935206    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:30.951289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.951257    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:26:30.961980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.961948    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:26:30.963452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.963405    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"]
Oct 30 20:26:30.994510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.991827    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:30.994510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.992291    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4ws\" (UniqueName: \"kubernetes.io/projected/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-kube-api-access-ml4ws\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:30.994510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:30.992657    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.093987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.093941    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4ws\" (UniqueName: \"kubernetes.io/projected/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-kube-api-access-ml4ws\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.094196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.094017    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.094196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.094141    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.094975 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.094941    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.096801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.096775    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.114418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.114393    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4ws\" (UniqueName: \"kubernetes.io/projected/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-kube-api-access-ml4ws\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.251636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.251559    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:31.571742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.571656    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"]
Oct 30 20:26:31.580027 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:31.579992    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76e8b49_f79e_4e7e_83fe_1f9f682aac79.slice/crio-43bf95349635095e3266b97145340c943d8b3dcd7ca04abf483c8b96fb17f3a0 WatchSource:0}: Error finding container 43bf95349635095e3266b97145340c943d8b3dcd7ca04abf483c8b96fb17f3a0: Status 404 returned error can't find the container with id 43bf95349635095e3266b97145340c943d8b3dcd7ca04abf483c8b96fb17f3a0
Oct 30 20:26:31.814939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:31.814898    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5" event={"ID":"e76e8b49-f79e-4e7e-83fe-1f9f682aac79","Type":"ContainerStarted","Data":"43bf95349635095e3266b97145340c943d8b3dcd7ca04abf483c8b96fb17f3a0"}
Oct 30 20:26:32.223574 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:26:32.223487    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:26:32.834021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:32.833984    4519 generic.go:356] "Generic (PLEG): container finished" podID="e76e8b49-f79e-4e7e-83fe-1f9f682aac79" containerID="b2da3b0de192e76a7f24e997580e156292c9d15e7284cedae20e4e5f5b3279ef" exitCode=0
Oct 30 20:26:32.834429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:32.834050    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5" event={"ID":"e76e8b49-f79e-4e7e-83fe-1f9f682aac79","Type":"ContainerDied","Data":"b2da3b0de192e76a7f24e997580e156292c9d15e7284cedae20e4e5f5b3279ef"}
Oct 30 20:26:34.090722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.090682    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:34.137320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.137229    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-kube-api-access-ml4ws\" (UniqueName: \"kubernetes.io/projected/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-kube-api-access-ml4ws\") pod \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") "
Oct 30 20:26:34.137320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.137277    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-openshift-ca\") pod \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") "
Oct 30 20:26:34.137598 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.137573    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-xks-ca\") pod \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\" (UID: \"e76e8b49-f79e-4e7e-83fe-1f9f682aac79\") "
Oct 30 20:26:34.138637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.138586    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-openshift-ca" pod "e76e8b49-f79e-4e7e-83fe-1f9f682aac79" (UID: "e76e8b49-f79e-4e7e-83fe-1f9f682aac79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:34.140050 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.140026    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-xks-ca" pod "e76e8b49-f79e-4e7e-83fe-1f9f682aac79" (UID: "e76e8b49-f79e-4e7e-83fe-1f9f682aac79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:34.140545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.140248    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-kube-api-access-ml4ws" pod "e76e8b49-f79e-4e7e-83fe-1f9f682aac79" (UID: "e76e8b49-f79e-4e7e-83fe-1f9f682aac79"). InnerVolumeSpecName "kube-api-access-ml4ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:34.320489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.320250    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:34.320489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.320295    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4ws\" (UniqueName: \"kubernetes.io/projected/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-kube-api-access-ml4ws\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:34.320489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.320311    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76e8b49-f79e-4e7e-83fe-1f9f682aac79-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:34.861197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.861160    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5" event={"ID":"e76e8b49-f79e-4e7e-83fe-1f9f682aac79","Type":"ContainerDied","Data":"43bf95349635095e3266b97145340c943d8b3dcd7ca04abf483c8b96fb17f3a0"}
Oct 30 20:26:34.861197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.861192    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"
Oct 30 20:26:34.861456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:34.861212    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43bf95349635095e3266b97145340c943d8b3dcd7ca04abf483c8b96fb17f3a0"
Oct 30 20:26:40.844027 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:40.843984    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"]
Oct 30 20:26:40.856223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:40.856195    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a3426-xwz96"]
Oct 30 20:26:41.614037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:41.614004    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56483207-d997-447f-831e-5906fe8164c5" path="/var/lib/kubelet/pods/56483207-d997-447f-831e-5906fe8164c5/volumes"
Oct 30 20:26:44.960119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.960076    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"]
Oct 30 20:26:44.960967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.960951    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e76e8b49-f79e-4e7e-83fe-1f9f682aac79" containerName="revoke-keys"
Oct 30 20:26:44.961029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.960970    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76e8b49-f79e-4e7e-83fe-1f9f682aac79" containerName="revoke-keys"
Oct 30 20:26:44.961104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.961092    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76e8b49-f79e-4e7e-83fe-1f9f682aac79" containerName="revoke-keys"
Oct 30 20:26:44.965590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.965567    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:44.985862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.985831    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"]
Oct 30 20:26:44.991246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.990372    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c19563-dockercfg-z5x49\""
Oct 30 20:26:44.991246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:44.991064    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c19563\""
Oct 30 20:26:45.029977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.029940    4519 scope.go:122] "RemoveContainer" containerID="dc42e71a54e3483d8d8dab770089d8c0805b36d323ee4ab7d9082753d2424494"
Oct 30 20:26:45.041346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.041322    4519 scope.go:122] "RemoveContainer" containerID="80af9f9d7add2184c59e30ec5fb08da702ccd6ce664cb6e80cc01057fd657fff"
Oct 30 20:26:45.119462 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.119427    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6571859c-a145-44d4-bbb0-e32ed236cef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:45.119660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.119607    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qs9n\" (UniqueName: \"kubernetes.io/projected/6571859c-a145-44d4-bbb0-e32ed236cef0-kube-api-access-7qs9n\") pod \"payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:45.220655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.220557    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6571859c-a145-44d4-bbb0-e32ed236cef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:45.220850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.220660    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qs9n\" (UniqueName: \"kubernetes.io/projected/6571859c-a145-44d4-bbb0-e32ed236cef0-kube-api-access-7qs9n\") pod \"payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:45.221356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.221332    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6571859c-a145-44d4-bbb0-e32ed236cef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:45.241718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.241684    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qs9n\" (UniqueName: \"kubernetes.io/projected/6571859c-a145-44d4-bbb0-e32ed236cef0-kube-api-access-7qs9n\") pod \"payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:45.285980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.285939    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:45.807706 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:45.798783    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6571859c_a145_44d4_bbb0_e32ed236cef0.slice/crio-faf2b4f25b0ada3ad7856a1592fda8377206487c79c52bff1805644e1724f2af WatchSource:0}: Error finding container faf2b4f25b0ada3ad7856a1592fda8377206487c79c52bff1805644e1724f2af: Status 404 returned error can't find the container with id faf2b4f25b0ada3ad7856a1592fda8377206487c79c52bff1805644e1724f2af
Oct 30 20:26:45.868123 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.866024    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"]
Oct 30 20:26:45.959548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:45.958709    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z" event={"ID":"6571859c-a145-44d4-bbb0-e32ed236cef0","Type":"ContainerStarted","Data":"faf2b4f25b0ada3ad7856a1592fda8377206487c79c52bff1805644e1724f2af"}
Oct 30 20:26:46.117253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.117202    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"]
Oct 30 20:26:46.127527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.125580    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6c4dd1-bhndv"]
Oct 30 20:26:46.398184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.375913    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"]
Oct 30 20:26:46.405430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.405399    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.407434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.407408    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"]
Oct 30 20:26:46.412456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.412269    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-model-cache\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.412714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.412690    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-tmp-dir\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.412919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.412903    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a97e1f-f884-4434-8104-2acb497c70f7-tls-certs\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.413129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.413115    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-dshm\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.413646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.413626    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-home\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.413828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.413810    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5md\" (UniqueName: \"kubernetes.io/projected/44a97e1f-f884-4434-8104-2acb497c70f7-kube-api-access-qj5md\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.435522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.432441    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"openshift-service-ca.crt\""
Oct 30 20:26:46.435522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.432555    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"kube-root-ca.crt\""
Oct 30 20:26:46.435522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.432864    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"default-dockercfg-x46lw\""
Oct 30 20:26:46.435522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.433000    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"rate-test-model-5c875d-kserve-self-signed-certs\""
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.515153    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a97e1f-f884-4434-8104-2acb497c70f7-tls-certs\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.515258    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-dshm\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.515299    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-home\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.515353    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5md\" (UniqueName: \"kubernetes.io/projected/44a97e1f-f884-4434-8104-2acb497c70f7-kube-api-access-qj5md\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.515578    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-model-cache\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.515622    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-tmp-dir\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.516094    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-model-cache\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.516165    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-home\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.516309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.516263    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-tmp-dir\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.518273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.518248    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-dshm\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.518565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.518547    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a97e1f-f884-4434-8104-2acb497c70f7-tls-certs\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.569489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.548945    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5md\" (UniqueName: \"kubernetes.io/projected/44a97e1f-f884-4434-8104-2acb497c70f7-kube-api-access-qj5md\") pod \"rate-test-model-5c875d-kserve-647897cc77-87fpl\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.731520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.731480    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:46.973722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.971992    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z" event={"ID":"6571859c-a145-44d4-bbb0-e32ed236cef0","Type":"ContainerStarted","Data":"63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0"}
Oct 30 20:26:46.973722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:46.973223    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:26:47.018576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:47.014285    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z" podStartSLOduration=3.014267212 podStartE2EDuration="3.014267212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:47.013171138 +0000 UTC m=+3131.513874862" watchObservedRunningTime="2026-10-30 20:26:47.014267212 +0000 UTC m=+3131.514970936"
Oct 30 20:26:47.139173 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:47.134501    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44a97e1f_f884_4434_8104_2acb497c70f7.slice/crio-c416b3e514aafa67e6b0476791a8f3d4bec780bbda83d3ed25b6342638f8231f WatchSource:0}: Error finding container c416b3e514aafa67e6b0476791a8f3d4bec780bbda83d3ed25b6342638f8231f: Status 404 returned error can't find the container with id c416b3e514aafa67e6b0476791a8f3d4bec780bbda83d3ed25b6342638f8231f
Oct 30 20:26:47.152740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:47.152701    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"]
Oct 30 20:26:47.613579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:47.613541    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd74ffd0-9048-4dac-96b2-d14f5ba3703a" path="/var/lib/kubelet/pods/dd74ffd0-9048-4dac-96b2-d14f5ba3703a/volumes"
Oct 30 20:26:47.983657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:47.983608    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl" event={"ID":"44a97e1f-f884-4434-8104-2acb497c70f7","Type":"ContainerStarted","Data":"383f656bb9c2121a0bdbe4c25e1f8bc1aff32dc0e70cb654fe5f2aff5d3e1852"}
Oct 30 20:26:47.983858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:47.983671    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl" event={"ID":"44a97e1f-f884-4434-8104-2acb497c70f7","Type":"ContainerStarted","Data":"c416b3e514aafa67e6b0476791a8f3d4bec780bbda83d3ed25b6342638f8231f"}
Oct 30 20:26:48.028101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:48.028047    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl" podStartSLOduration=2.028029708 podStartE2EDuration="2.028029708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:48.024898333 +0000 UTC m=+3132.525602059" watchObservedRunningTime="2026-10-30 20:26:48.028029708 +0000 UTC m=+3132.528733435"
Oct 30 20:26:48.416382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:48.416339    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"]
Oct 30 20:26:48.416851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:48.416677    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" containerID="cri-o://08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded" gracePeriod=30
Oct 30 20:26:48.732448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:48.732372    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:51.865213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.865164    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"]
Oct 30 20:26:51.872150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.872113    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:51.896815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.896775    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"]
Oct 30 20:26:51.908299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.908266    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"kube-root-ca.crt\""
Oct 30 20:26:51.908561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.908287    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"openshift-service-ca.crt\""
Oct 30 20:26:51.909287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.909263    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"rate-test-model-9d2720-kserve-self-signed-certs\""
Oct 30 20:26:51.909287 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.909264    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d2720\"/\"default-dockercfg-6dq9g\""
Oct 30 20:26:51.931189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.931155    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tls-certs\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:51.931351 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.931199    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-home\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:51.931351 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.931337    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tmp-dir\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:51.931542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.931526    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-model-cache\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:51.931616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.931573    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bxs\" (UniqueName: \"kubernetes.io/projected/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-kube-api-access-p4bxs\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:51.931707 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:51.931691    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-dshm\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.032759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.032723    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tls-certs\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.032759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.032765    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-home\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.033021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.032821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tmp-dir\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.033021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.032924    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-model-cache\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.033021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.032956    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bxs\" (UniqueName: \"kubernetes.io/projected/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-kube-api-access-p4bxs\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.033021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.033018    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-dshm\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.033326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.033293    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-home\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.033421 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.033345    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tmp-dir\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.033421 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.033389    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-model-cache\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.035539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.035512    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-dshm\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.035650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.035540    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tls-certs\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.058119 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.056601    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bxs\" (UniqueName: \"kubernetes.io/projected/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-kube-api-access-p4bxs\") pod \"rate-test-model-9d2720-kserve-86b49859cc-sd6hr\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") " pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.188540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.188451    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:52.266491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.265743    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:26:52.266491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.265816    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:26:52.603410 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:26:52.603369    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63dd0b1b_6da2_4999_8b21_2f4dd73a15c0.slice/crio-b540c54f1570136d9771de43339e1de33c2619326c1b11f9fad3c4e6b8ce0bcb WatchSource:0}: Error finding container b540c54f1570136d9771de43339e1de33c2619326c1b11f9fad3c4e6b8ce0bcb: Status 404 returned error can't find the container with id b540c54f1570136d9771de43339e1de33c2619326c1b11f9fad3c4e6b8ce0bcb
Oct 30 20:26:52.605168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:52.605042    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"]
Oct 30 20:26:53.028345 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.028298    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr" event={"ID":"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0","Type":"ContainerStarted","Data":"fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3"}
Oct 30 20:26:53.028774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.028356    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr" event={"ID":"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0","Type":"ContainerStarted","Data":"b540c54f1570136d9771de43339e1de33c2619326c1b11f9fad3c4e6b8ce0bcb"}
Oct 30 20:26:53.076522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.075098    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr" podStartSLOduration=2.075082394 podStartE2EDuration="2.075082394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:53.07405173 +0000 UTC m=+3137.574755455" watchObservedRunningTime="2026-10-30 20:26:53.075082394 +0000 UTC m=+3137.575786118"
Oct 30 20:26:53.189145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.189092    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:26:53.743863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.743834    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 20:26:53.887934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.887901    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-envoy\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.887943    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-podinfo\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.887962    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-data\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888123    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-socket\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888183    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-token\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-token\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888240    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-credential-socket\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888348    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-kube-api-access-8g6w9\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-kube-api-access-8g6w9\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888360    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-socket" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:53.888500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888482    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-certs\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888542    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istiod-ca-cert\") pod \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\" (UID: \"ea0509ee-c53f-4ea0-a1f4-d50be64daf27\") "
Oct 30 20:26:53.888654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888553    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-credential-socket" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:53.888804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888781    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-certs" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:53.888911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.888295    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-data" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:53.889189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.889131    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istiod-ca-cert" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:53.889320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.889300    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.889387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.889326    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.889387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.889340    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.889387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.889354    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.889387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.889367    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.890621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.890593    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-podinfo" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:53.890741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.890589    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-token" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:53.890741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.890705    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-envoy" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:53.890858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.890754    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-kube-api-access-8g6w9" pod "ea0509ee-c53f-4ea0-a1f4-d50be64daf27" (UID: "ea0509ee-c53f-4ea0-a1f4-d50be64daf27"). InnerVolumeSpecName "kube-api-access-8g6w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:53.991322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.991224    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.991322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.991262    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.991322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.991271    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:53.991322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:53.991281    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6w9\" (UniqueName: \"kubernetes.io/projected/ea0509ee-c53f-4ea0-a1f4-d50be64daf27-kube-api-access-8g6w9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:26:54.038375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.038336    4519 generic.go:356] "Generic (PLEG): container finished" podID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerID="08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded" exitCode=0
Oct 30 20:26:54.038849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.038388    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"
Oct 30 20:26:54.038849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.038422    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" event={"ID":"ea0509ee-c53f-4ea0-a1f4-d50be64daf27","Type":"ContainerDied","Data":"08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded"}
Oct 30 20:26:54.038849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.038506    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr" event={"ID":"ea0509ee-c53f-4ea0-a1f4-d50be64daf27","Type":"ContainerDied","Data":"560703f8054edc2188b3afddb9c86bb6a6a3493dd35598587922cad8f4ff18db"}
Oct 30 20:26:54.038849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.038540    4519 scope.go:122] "RemoveContainer" containerID="08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded"
Oct 30 20:26:54.058631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.058606    4519 scope.go:122] "RemoveContainer" containerID="08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded"
Oct 30 20:26:54.058953 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:26:54.058930    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded\": container with ID starting with 08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded not found: ID does not exist" containerID="08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded"
Oct 30 20:26:54.059252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.058965    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded"} err="failed to get container status \"08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded\": rpc error: code = NotFound desc = could not find container \"08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded\": container with ID starting with 08b50f99ef28ccada111ba79f418bec730ea9f3598fbeda1ba99687d8fda8ded not found: ID does not exist"
Oct 30 20:26:54.111302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.111264    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"]
Oct 30 20:26:54.125054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:54.125012    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-cptcr"]
Oct 30 20:26:55.613093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:55.613052    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" path="/var/lib/kubelet/pods/ea0509ee-c53f-4ea0-a1f4-d50be64daf27/volumes"
Oct 30 20:26:56.732358 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:56.732314    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:56.746008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:56.745976    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:57.085402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:57.085301    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:26:58.995346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:26:58.995311    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:27:00.632028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.631993    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"]
Oct 30 20:27:00.632665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.632648    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy"
Oct 30 20:27:00.632665 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.632666    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy"
Oct 30 20:27:00.632802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.632793    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0509ee-c53f-4ea0-a1f4-d50be64daf27" containerName="istio-proxy"
Oct 30 20:27:00.690952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.690918    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"]
Oct 30 20:27:00.691395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.691373    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.718282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.718249    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-843ea8-openshift-default-dockercfg-mvxps\""
Oct 30 20:27:00.836460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.836415    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-certs\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.860494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.836768    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7pld\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-kube-api-access-r7pld\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.860853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.860819    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-data\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.861033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.861013    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29106e39-77cd-4a3c-8766-984760dc28ad-istiod-ca-cert\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.861189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.861170    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-credential-socket\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.861374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.861358    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-socket\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.861588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.861567    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-istio-token\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.861729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.861713    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29106e39-77cd-4a3c-8766-984760dc28ad-istio-podinfo\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.861916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.861900    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-envoy\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.972310    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-socket\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.972871    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-socket\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.973862    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-istio-token\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.973898    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29106e39-77cd-4a3c-8766-984760dc28ad-istio-podinfo\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.973981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-envoy\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.974229    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-certs\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.974338    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7pld\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-kube-api-access-r7pld\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.974379    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-data\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.974428    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29106e39-77cd-4a3c-8766-984760dc28ad-istiod-ca-cert\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.974493    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-credential-socket\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.975075    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-certs\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.976496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.975671    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-data\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.977291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.977084    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29106e39-77cd-4a3c-8766-984760dc28ad-istiod-ca-cert\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.977749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.977721    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-credential-socket\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.988384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.988108    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29106e39-77cd-4a3c-8766-984760dc28ad-istio-podinfo\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.988384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.988337    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-envoy\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.999282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.999233    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7pld\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-kube-api-access-r7pld\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:00.999909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:00.999881    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-istio-token\") pod \"e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") " pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:01.018978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:01.018941    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:01.374015 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:27:01.373966    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29106e39_77cd_4a3c_8766_984760dc28ad.slice/crio-22a754b43c68f370cd92fbd1af643504c8bde9ff7e18cb7326635a855a5c7bc1 WatchSource:0}: Error finding container 22a754b43c68f370cd92fbd1af643504c8bde9ff7e18cb7326635a855a5c7bc1: Status 404 returned error can't find the container with id 22a754b43c68f370cd92fbd1af643504c8bde9ff7e18cb7326635a855a5c7bc1
Oct 30 20:27:01.376309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:01.376280    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"]
Oct 30 20:27:01.377412 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:01.377374    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:27:01.377564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:01.377457    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:27:01.377564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:01.377525    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:27:02.188808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:02.188766    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:27:02.190227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:02.190187    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" event={"ID":"29106e39-77cd-4a3c-8766-984760dc28ad","Type":"ContainerStarted","Data":"5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0"}
Oct 30 20:27:02.190371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:02.190248    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" event={"ID":"29106e39-77cd-4a3c-8766-984760dc28ad","Type":"ContainerStarted","Data":"22a754b43c68f370cd92fbd1af643504c8bde9ff7e18cb7326635a855a5c7bc1"}
Oct 30 20:27:02.204052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:02.204022    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:27:02.227013 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:27:02.226980    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:27:02.239728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:02.239678    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podStartSLOduration=2.239664991 podStartE2EDuration="2.239664991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:02.235618385 +0000 UTC m=+3146.736322110" watchObservedRunningTime="2026-10-30 20:27:02.239664991 +0000 UTC m=+3146.740368714"
Oct 30 20:27:03.019323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:03.019277    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:03.026245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:03.026219    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:03.200319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:03.198760    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:03.200319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:03.199927    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:27:03.203654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:03.203622    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:27:11.108065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.108023    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"]
Oct 30 20:27:11.108553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.108290    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p" podUID="3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" containerName="payload-processing" containerID="cri-o://77f63be8be6d2a8435a76504bb22fddd764a2cf318b32a1c0db0bfca3d3fa028" gracePeriod=30
Oct 30 20:27:11.323023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.322665    4519 generic.go:356] "Generic (PLEG): container finished" podID="3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" containerID="77f63be8be6d2a8435a76504bb22fddd764a2cf318b32a1c0db0bfca3d3fa028" exitCode=0
Oct 30 20:27:11.323877 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.323285    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p" event={"ID":"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e","Type":"ContainerDied","Data":"77f63be8be6d2a8435a76504bb22fddd764a2cf318b32a1c0db0bfca3d3fa028"}
Oct 30 20:27:11.675709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.675678    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:27:11.749507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.749460    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-plugins-config-volume\") pod \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") "
Oct 30 20:27:11.749737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.749712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-kube-api-access-6hnrn\" (UniqueName: \"kubernetes.io/projected/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-kube-api-access-6hnrn\") pod \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\" (UID: \"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e\") "
Oct 30 20:27:11.749953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.749931    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-plugins-config-volume" pod "3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" (UID: "3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:11.750387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.750358    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:11.752443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.752401    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-kube-api-access-6hnrn" pod "3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" (UID: "3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e"). InnerVolumeSpecName "kube-api-access-6hnrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:11.833353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.833302    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"]
Oct 30 20:27:11.838162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.838128    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-188d45-k8rn6"]
Oct 30 20:27:11.851666 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:11.851628    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hnrn\" (UniqueName: \"kubernetes.io/projected/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e-kube-api-access-6hnrn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:12.334067 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:12.334030    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"
Oct 30 20:27:12.334578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:12.334061    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p" event={"ID":"3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e","Type":"ContainerDied","Data":"6a0d30306f37a44589ce8e5a902b2a1c4071662e78e4086e6e45df1b4096d51a"}
Oct 30 20:27:12.334578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:12.334126    4519 scope.go:122] "RemoveContainer" containerID="77f63be8be6d2a8435a76504bb22fddd764a2cf318b32a1c0db0bfca3d3fa028"
Oct 30 20:27:12.387285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:12.387200    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"]
Oct 30 20:27:12.398832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:12.398798    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-64bfd679b6-ffn9p"]
Oct 30 20:27:13.612968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:13.612931    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" path="/var/lib/kubelet/pods/3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e/volumes"
Oct 30 20:27:13.613434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:13.613419    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860cdd40-231b-48b5-866e-cd2700dd183c" path="/var/lib/kubelet/pods/860cdd40-231b-48b5-866e-cd2700dd183c/volumes"
Oct 30 20:27:17.281933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:17.281890    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"]
Oct 30 20:27:19.327960 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:19.327920    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"]
Oct 30 20:27:19.339205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:19.339175    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbda3-bmvb5"]
Oct 30 20:27:19.612660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:19.612583    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76e8b49-f79e-4e7e-83fe-1f9f682aac79" path="/var/lib/kubelet/pods/e76e8b49-f79e-4e7e-83fe-1f9f682aac79/volumes"
Oct 30 20:27:26.372349 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.372306    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"]
Oct 30 20:27:26.373027 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.372970    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy" containerID="cri-o://bfdd5282829684254f1c5043db656528f671cb3f737ac15a703ae406d6546dcf" gracePeriod=30
Oct 30 20:27:26.921208 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.921170    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"]
Oct 30 20:27:26.921762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.921747    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" containerName="payload-processing"
Oct 30 20:27:26.921813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.921763    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" containerName="payload-processing"
Oct 30 20:27:26.921903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.921894    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9bf1a0-d8ce-4c79-b412-8c9852cd6e9e" containerName="payload-processing"
Oct 30 20:27:26.925619 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.925601    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:26.956242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:26.956208    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"]
Oct 30 20:27:27.007545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.007513    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:27:27.007545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.007527    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:27:27.047004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.046970    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.047331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.047307    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.047428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.047366    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbcq7\" (UniqueName: \"kubernetes.io/projected/3298c124-63ac-4ccd-87f2-98002a466cd7-kube-api-access-vbcq7\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.148274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.148237    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.148531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.148507    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.148628 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.148576    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbcq7\" (UniqueName: \"kubernetes.io/projected/3298c124-63ac-4ccd-87f2-98002a466cd7-kube-api-access-vbcq7\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.149364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.149340    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.151678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.151653    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.167555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.167533    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbcq7\" (UniqueName: \"kubernetes.io/projected/3298c124-63ac-4ccd-87f2-98002a466cd7-kube-api-access-vbcq7\") pod \"maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.240559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.240443    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:27.556178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:27.556144    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"]
Oct 30 20:27:28.566109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:28.566065    4519 generic.go:356] "Generic (PLEG): container finished" podID="3298c124-63ac-4ccd-87f2-98002a466cd7" containerID="badb0ae4c693b2ee08ab8611fd91d97ba50213b1c8a1df74f014cbeb38b29fba" exitCode=0
Oct 30 20:27:28.566626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:28.566141    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr" event={"ID":"3298c124-63ac-4ccd-87f2-98002a466cd7","Type":"ContainerDied","Data":"badb0ae4c693b2ee08ab8611fd91d97ba50213b1c8a1df74f014cbeb38b29fba"}
Oct 30 20:27:28.566626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:28.566203    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr" event={"ID":"3298c124-63ac-4ccd-87f2-98002a466cd7","Type":"ContainerStarted","Data":"cef389b3fa88ba1aed3ac378c6703712d0c77d93f69825c27ff5a049a81c59bf"}
Oct 30 20:27:29.770440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.770414    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:29.818387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.818358    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-xks-ca\") pod \"3298c124-63ac-4ccd-87f2-98002a466cd7\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") "
Oct 30 20:27:29.818605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.818583    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3298c124-63ac-4ccd-87f2-98002a466cd7-kube-api-access-vbcq7\" (UniqueName: \"kubernetes.io/projected/3298c124-63ac-4ccd-87f2-98002a466cd7-kube-api-access-vbcq7\") pod \"3298c124-63ac-4ccd-87f2-98002a466cd7\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") "
Oct 30 20:27:29.818686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.818633    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-openshift-ca\") pod \"3298c124-63ac-4ccd-87f2-98002a466cd7\" (UID: \"3298c124-63ac-4ccd-87f2-98002a466cd7\") "
Oct 30 20:27:29.819649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.819310    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-openshift-ca" pod "3298c124-63ac-4ccd-87f2-98002a466cd7" (UID: "3298c124-63ac-4ccd-87f2-98002a466cd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:29.821316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.821291    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-xks-ca" pod "3298c124-63ac-4ccd-87f2-98002a466cd7" (UID: "3298c124-63ac-4ccd-87f2-98002a466cd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:29.821436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.821307    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3298c124-63ac-4ccd-87f2-98002a466cd7-kube-api-access-vbcq7" pod "3298c124-63ac-4ccd-87f2-98002a466cd7" (UID: "3298c124-63ac-4ccd-87f2-98002a466cd7"). InnerVolumeSpecName "kube-api-access-vbcq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:29.920333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.920293    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:29.920333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.920325    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbcq7\" (UniqueName: \"kubernetes.io/projected/3298c124-63ac-4ccd-87f2-98002a466cd7-kube-api-access-vbcq7\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:29.920557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:29.920341    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3298c124-63ac-4ccd-87f2-98002a466cd7-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:30.591272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:30.591229    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"
Oct 30 20:27:30.591272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:30.591248    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr" event={"ID":"3298c124-63ac-4ccd-87f2-98002a466cd7","Type":"ContainerDied","Data":"cef389b3fa88ba1aed3ac378c6703712d0c77d93f69825c27ff5a049a81c59bf"}
Oct 30 20:27:30.591552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:30.591345    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef389b3fa88ba1aed3ac378c6703712d0c77d93f69825c27ff5a049a81c59bf"
Oct 30 20:27:31.609059 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.609020    4519 generic.go:356] "Generic (PLEG): container finished" podID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerID="bfdd5282829684254f1c5043db656528f671cb3f737ac15a703ae406d6546dcf" exitCode=0
Oct 30 20:27:31.614531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.614446    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" event={"ID":"ee3085fd-c4f8-431f-a214-6ad1bfd1288e","Type":"ContainerDied","Data":"bfdd5282829684254f1c5043db656528f671cb3f737ac15a703ae406d6546dcf"}
Oct 30 20:27:31.614710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.614576    4519 scope.go:122] "RemoveContainer" containerID="5baf4cdf932b7509d8bf981fce56eb327804561cf15aef3a2a1fd55637dfc400"
Oct 30 20:27:31.677459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.677430    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.744556    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-token\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-token\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.744629    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-credential-socket\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.744828    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-kube-api-access-g7vkx\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-kube-api-access-g7vkx\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.744930    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-podinfo\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.744988    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-data\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.745051    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-envoy\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.745126    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-certs\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.745222    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-socket\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.745261    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istiod-ca-cert\") pod \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\" (UID: \"ee3085fd-c4f8-431f-a214-6ad1bfd1288e\") "
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.745802    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-credential-socket" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.745936    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.746489    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istiod-ca-cert" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.746669    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-certs" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.746757    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-socket" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:31.747878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.746783    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-data" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:31.749646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.749609    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-envoy" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:31.749979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.749955    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-kube-api-access-g7vkx" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "kube-api-access-g7vkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:31.751622 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.751600    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-token" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:31.767017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.766983    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-podinfo" pod "ee3085fd-c4f8-431f-a214-6ad1bfd1288e" (UID: "ee3085fd-c4f8-431f-a214-6ad1bfd1288e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848358    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848398    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848413    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848426    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848440    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848453    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848507    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7vkx\" (UniqueName: \"kubernetes.io/projected/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-kube-api-access-g7vkx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:31.848553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:31.848526    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3085fd-c4f8-431f-a214-6ad1bfd1288e-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:32.226760 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:27:32.226722    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:27:32.303488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.303427    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l" podUID="8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" containerName="main" containerID="cri-o://06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9" gracePeriod=45
Oct 30 20:27:32.595591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.595524    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:27:32.621938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.621903    4519 generic.go:356] "Generic (PLEG): container finished" podID="8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" containerID="06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9" exitCode=0
Oct 30 20:27:32.622360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.621939    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"
Oct 30 20:27:32.622360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.621982    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l" event={"ID":"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2","Type":"ContainerDied","Data":"06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9"}
Oct 30 20:27:32.622360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.622024    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l" event={"ID":"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2","Type":"ContainerDied","Data":"44d97fb2e1cd17d7af9b36ec8f8d08582b1a600cba54b91c21cfd0770d873786"}
Oct 30 20:27:32.622360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.622046    4519 scope.go:122] "RemoveContainer" containerID="06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9"
Oct 30 20:27:32.632297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.632264    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg" event={"ID":"ee3085fd-c4f8-431f-a214-6ad1bfd1288e","Type":"ContainerDied","Data":"60e56dbcd77720b29f1f13d4d92f42149d36ed404b017b07a42ba5dc53eb9d7c"}
Oct 30 20:27:32.632412 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.632387    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"
Oct 30 20:27:32.633607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.633584    4519 scope.go:122] "RemoveContainer" containerID="06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9"
Oct 30 20:27:32.633887 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:27:32.633856    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9\": container with ID starting with 06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9 not found: ID does not exist" containerID="06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9"
Oct 30 20:27:32.634003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.633897    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9"} err="failed to get container status \"06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9\": rpc error: code = NotFound desc = could not find container \"06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9\": container with ID starting with 06e15da13e994396e8112ed72678ce1f2107104bb65278a6b24a9163032292f9 not found: ID does not exist"
Oct 30 20:27:32.634003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.633922    4519 scope.go:122] "RemoveContainer" containerID="bfdd5282829684254f1c5043db656528f671cb3f737ac15a703ae406d6546dcf"
Oct 30 20:27:32.660919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.660883    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-model-cache\") pod \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") "
Oct 30 20:27:32.661061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.660984    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tmp-dir\") pod \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") "
Oct 30 20:27:32.661148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.661095    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tls-certs\") pod \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") "
Oct 30 20:27:32.661210 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.661192    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-dshm\") pod \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") "
Oct 30 20:27:32.661380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.661283    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-kube-api-access-sfs5s\" (UniqueName: \"kubernetes.io/projected/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-kube-api-access-sfs5s\") pod \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") "
Oct 30 20:27:32.661445 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.661378    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-home\") pod \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\" (UID: \"8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2\") "
Oct 30 20:27:32.662765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.662735    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-model-cache" pod "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" (UID: "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:32.662966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.662941    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tmp-dir" pod "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" (UID: "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:32.663762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.663736    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-home" pod "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" (UID: "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:32.683928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.677871    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-dshm" pod "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" (UID: "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:32.683928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.679625    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tls-certs" pod "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" (UID: "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:32.698167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.698135    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-kube-api-access-sfs5s" pod "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" (UID: "8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2"). InnerVolumeSpecName "kube-api-access-sfs5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:32.715760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.715726    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"]
Oct 30 20:27:32.723236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.723209    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbda3-openshift-default-67f8cf4789-fg9pg"]
Oct 30 20:27:32.763338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.763253    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:32.763338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.763286    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:32.763338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.763296    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfs5s\" (UniqueName: \"kubernetes.io/projected/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-kube-api-access-sfs5s\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:32.763338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.763318    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:32.763338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.763329    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:32.763338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.763338    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:27:32.976050 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.976013    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"]
Oct 30 20:27:32.990244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:32.990197    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-188d45/ipp-route-188d45-kserve-c85cbfd85-wqd8l"]
Oct 30 20:27:33.615188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:33.615146    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" path="/var/lib/kubelet/pods/8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2/volumes"
Oct 30 20:27:33.615869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:27:33.615851    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" path="/var/lib/kubelet/pods/ee3085fd-c4f8-431f-a214-6ad1bfd1288e/volumes"
Oct 30 20:28:01.498021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:01.497967    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"]
Oct 30 20:28:01.503626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:01.499800    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy" containerID="cri-o://acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1" gracePeriod=30
Oct 30 20:28:02.222129 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:02.222090    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:28:06.764364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.764326    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"]
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765260    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" containerName="main"
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765284    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" containerName="main"
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765299    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3298c124-63ac-4ccd-87f2-98002a466cd7" containerName="revoke-keys"
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765308    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3298c124-63ac-4ccd-87f2-98002a466cd7" containerName="revoke-keys"
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765365    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy"
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765372    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy"
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765561    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy"
Oct 30 20:28:06.765587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765578    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy"
Oct 30 20:28:06.766069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765594    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8bce9e-0347-49ee-b0f0-ac1f2b0cfca2" containerName="main"
Oct 30 20:28:06.766069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.765605    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3298c124-63ac-4ccd-87f2-98002a466cd7" containerName="revoke-keys"
Oct 30 20:28:06.770231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.770211    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:06.781219 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.781189    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"]
Oct 30 20:28:06.786060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.786020    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:06.786181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.786068    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:28:06.824403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.824380    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:28:06.929821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.929787    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-envoy\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.929833    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-socket\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.929881    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85027075-443e-4323-a372-86b86a6dd098-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85027075-443e-4323-a372-86b86a6dd098-istiod-ca-cert\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.929966    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-data\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930001    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-kube-api-access-9nm2s\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-kube-api-access-9nm2s\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930029    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-credential-socket\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930057    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-certs\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930082    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-istio-token\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-istio-token\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930124    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85027075-443e-4323-a372-86b86a6dd098-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85027075-443e-4323-a372-86b86a6dd098-istio-podinfo\") pod \"85027075-443e-4323-a372-86b86a6dd098\" (UID: \"85027075-443e-4323-a372-86b86a6dd098\") "
Oct 30 20:28:06.930258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930234    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-socket" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.930556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930310    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-data" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.930556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930521    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-certs" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.930654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930558    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85027075-443e-4323-a372-86b86a6dd098-istiod-ca-cert" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:06.930997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930752    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-credential-socket" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.930997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.930917    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:06.931521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.931457    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:06.931660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.931628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7h7\" (UniqueName: \"kubernetes.io/projected/bea19d21-85d0-4d51-88cd-bacde4ebbe19-kube-api-access-pd7h7\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:06.931863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.931832    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:06.931907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.931871    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:06.931907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.931886    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:06.931907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.931902    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:06.932003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.931917    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85027075-443e-4323-a372-86b86a6dd098-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:06.932441 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.932415    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-envoy" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.933022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.933000    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-kube-api-access-9nm2s" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "kube-api-access-9nm2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:06.933234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.933205    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85027075-443e-4323-a372-86b86a6dd098-istio-podinfo" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:28:06.933330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:06.933248    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-istio-token" pod "85027075-443e-4323-a372-86b86a6dd098" (UID: "85027075-443e-4323-a372-86b86a6dd098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:07.002540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.002496    4519 generic.go:356] "Generic (PLEG): container finished" podID="85027075-443e-4323-a372-86b86a6dd098" containerID="acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1" exitCode=0
Oct 30 20:28:07.002734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.002580    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" event={"ID":"85027075-443e-4323-a372-86b86a6dd098","Type":"ContainerDied","Data":"acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1"}
Oct 30 20:28:07.002734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.002614    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"
Oct 30 20:28:07.002734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.002639    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz" event={"ID":"85027075-443e-4323-a372-86b86a6dd098","Type":"ContainerDied","Data":"0b6ea8033572d1b2b708940ec38543afad7b9908a2f7a0baebd77d7dff882a1c"}
Oct 30 20:28:07.002734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.002670    4519 scope.go:122] "RemoveContainer" containerID="acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1"
Oct 30 20:28:07.017386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.017346    4519 scope.go:122] "RemoveContainer" containerID="b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451"
Oct 30 20:28:07.027729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.027696    4519 scope.go:122] "RemoveContainer" containerID="acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1"
Oct 30 20:28:07.028021 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:07.027992    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1\": container with ID starting with acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1 not found: ID does not exist" containerID="acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1"
Oct 30 20:28:07.028108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.028033    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1"} err="failed to get container status \"acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1\": rpc error: code = NotFound desc = could not find container \"acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1\": container with ID starting with acb7d26c91990d6494cbd7c0ca86fe944b581520ad5b24286de74573499881b1 not found: ID does not exist"
Oct 30 20:28:07.028108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.028054    4519 scope.go:122] "RemoveContainer" containerID="b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451"
Oct 30 20:28:07.028327 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:07.028302    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451\": container with ID starting with b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451 not found: ID does not exist" containerID="b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451"
Oct 30 20:28:07.028389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.028342    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451"} err="failed to get container status \"b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451\": rpc error: code = NotFound desc = could not find container \"b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451\": container with ID starting with b5b19c630063ef90ff27c7f1e7d70e25b9741c2df17baad838be2c4b1709c451 not found: ID does not exist"
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.035401    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7h7\" (UniqueName: \"kubernetes.io/projected/bea19d21-85d0-4d51-88cd-bacde4ebbe19-kube-api-access-pd7h7\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.035709    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.035751    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.035850    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85027075-443e-4323-a372-86b86a6dd098-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.035875    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nm2s\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-kube-api-access-9nm2s\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.035892    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85027075-443e-4323-a372-86b86a6dd098-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.035906    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85027075-443e-4323-a372-86b86a6dd098-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:07.038165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.036644    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:07.039961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.039931    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:07.057565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.057527    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7h7\" (UniqueName: \"kubernetes.io/projected/bea19d21-85d0-4d51-88cd-bacde4ebbe19-kube-api-access-pd7h7\") pod \"maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:07.060511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.060476    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"]
Oct 30 20:28:07.067007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.066980    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-sbwpz"]
Oct 30 20:28:07.119791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.119763    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:07.439522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.439493    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"]
Oct 30 20:28:07.440240 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:28:07.440213    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea19d21_85d0_4d51_88cd_bacde4ebbe19.slice/crio-c34c22afdb9a84a2af95803a7d2ccacaa5cc276b476142bb1210804a8e4b9de2 WatchSource:0}: Error finding container c34c22afdb9a84a2af95803a7d2ccacaa5cc276b476142bb1210804a8e4b9de2: Status 404 returned error can't find the container with id c34c22afdb9a84a2af95803a7d2ccacaa5cc276b476142bb1210804a8e4b9de2
Oct 30 20:28:07.613156 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.613122    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85027075-443e-4323-a372-86b86a6dd098" path="/var/lib/kubelet/pods/85027075-443e-4323-a372-86b86a6dd098/volumes"
Oct 30 20:28:07.891531 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.891492    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"]
Oct 30 20:28:07.892018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.891753    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg" podUID="ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" containerName="payload-processing" containerID="cri-o://ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3" gracePeriod=30
Oct 30 20:28:07.930569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.930521    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"]
Oct 30 20:28:07.930828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:07.930795    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq" podUID="5e5fdd18-17f4-4aeb-9ce3-42e134996662" containerName="payload-pre-processing" containerID="cri-o://87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6" gracePeriod=30
Oct 30 20:28:08.025561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.021991    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp" event={"ID":"bea19d21-85d0-4d51-88cd-bacde4ebbe19","Type":"ContainerStarted","Data":"e868961abf9817e22fa2ce96a1c948a8a1c7d8f12cb9f8679fa35d974533b03c"}
Oct 30 20:28:08.025561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.022059    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp" event={"ID":"bea19d21-85d0-4d51-88cd-bacde4ebbe19","Type":"ContainerStarted","Data":"c34c22afdb9a84a2af95803a7d2ccacaa5cc276b476142bb1210804a8e4b9de2"}
Oct 30 20:28:08.071049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.070986    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp" podStartSLOduration=2.070966384 podStartE2EDuration="2.070966384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:08.063334245 +0000 UTC m=+3212.564037970" watchObservedRunningTime="2026-10-30 20:28:08.070966384 +0000 UTC m=+3212.571670108"
Oct 30 20:28:08.407757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.407361    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:28:08.477102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.476343    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-plugins-config-volume\") pod \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") "
Oct 30 20:28:08.477102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.476396    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-kube-api-access-zhtxm\" (UniqueName: \"kubernetes.io/projected/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-kube-api-access-zhtxm\") pod \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\" (UID: \"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30\") "
Oct 30 20:28:08.477989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.477949    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-plugins-config-volume" pod "ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" (UID: "ddbdb14d-001a-4e7b-a922-2ecbe27f3f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:08.478482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.478447    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:28:08.482122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.480456    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-kube-api-access-zhtxm" pod "ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" (UID: "ddbdb14d-001a-4e7b-a922-2ecbe27f3f30"). InnerVolumeSpecName "kube-api-access-zhtxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:08.577784 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.577754    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5fdd18-17f4-4aeb-9ce3-42e134996662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5fdd18-17f4-4aeb-9ce3-42e134996662-plugins-config-volume\") pod \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") "
Oct 30 20:28:08.577958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.577904    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5fdd18-17f4-4aeb-9ce3-42e134996662-kube-api-access-tptxh\" (UniqueName: \"kubernetes.io/projected/5e5fdd18-17f4-4aeb-9ce3-42e134996662-kube-api-access-tptxh\") pod \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\" (UID: \"5e5fdd18-17f4-4aeb-9ce3-42e134996662\") "
Oct 30 20:28:08.578225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.578182    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5fdd18-17f4-4aeb-9ce3-42e134996662-plugins-config-volume" pod "5e5fdd18-17f4-4aeb-9ce3-42e134996662" (UID: "5e5fdd18-17f4-4aeb-9ce3-42e134996662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:08.578443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.578423    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:08.578530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.578446    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtxm\" (UniqueName: \"kubernetes.io/projected/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30-kube-api-access-zhtxm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:08.578530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.578456    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5fdd18-17f4-4aeb-9ce3-42e134996662-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:08.580155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.580129    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5fdd18-17f4-4aeb-9ce3-42e134996662-kube-api-access-tptxh" pod "5e5fdd18-17f4-4aeb-9ce3-42e134996662" (UID: "5e5fdd18-17f4-4aeb-9ce3-42e134996662"). InnerVolumeSpecName "kube-api-access-tptxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:08.679404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.679369    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tptxh\" (UniqueName: \"kubernetes.io/projected/5e5fdd18-17f4-4aeb-9ce3-42e134996662-kube-api-access-tptxh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:08.809353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.809310    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"]
Oct 30 20:28:08.819232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:08.819203    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f35cc1-f2jvr"]
Oct 30 20:28:09.033996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.033953    4519 generic.go:356] "Generic (PLEG): container finished" podID="5e5fdd18-17f4-4aeb-9ce3-42e134996662" containerID="87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6" exitCode=0
Oct 30 20:28:09.034639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.033998    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq" event={"ID":"5e5fdd18-17f4-4aeb-9ce3-42e134996662","Type":"ContainerDied","Data":"87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6"}
Oct 30 20:28:09.034744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.034041    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"
Oct 30 20:28:09.034744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.034690    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq" event={"ID":"5e5fdd18-17f4-4aeb-9ce3-42e134996662","Type":"ContainerDied","Data":"f8cf5fe746c41965ce84a23abedd623f838dbc46981fdc02b9bcda49eafa532a"}
Oct 30 20:28:09.034744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.034734    4519 scope.go:122] "RemoveContainer" containerID="87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6"
Oct 30 20:28:09.063585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.050871    4519 generic.go:356] "Generic (PLEG): container finished" podID="ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" containerID="ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3" exitCode=0
Oct 30 20:28:09.063585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.050969    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg" event={"ID":"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30","Type":"ContainerDied","Data":"ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3"}
Oct 30 20:28:09.063585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.051017    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg" event={"ID":"ddbdb14d-001a-4e7b-a922-2ecbe27f3f30","Type":"ContainerDied","Data":"d53f69ffee8dd668b21a1e95f518adc51095aa4b7f3996765e5ab9f6a0d17f64"}
Oct 30 20:28:09.063585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.051103    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"
Oct 30 20:28:09.123839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.123559    4519 generic.go:356] "Generic (PLEG): container finished" podID="bea19d21-85d0-4d51-88cd-bacde4ebbe19" containerID="e868961abf9817e22fa2ce96a1c948a8a1c7d8f12cb9f8679fa35d974533b03c" exitCode=0
Oct 30 20:28:09.123839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.123743    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp" event={"ID":"bea19d21-85d0-4d51-88cd-bacde4ebbe19","Type":"ContainerDied","Data":"e868961abf9817e22fa2ce96a1c948a8a1c7d8f12cb9f8679fa35d974533b03c"}
Oct 30 20:28:09.166511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.161456    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"]
Oct 30 20:28:09.182052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.182024    4519 scope.go:122] "RemoveContainer" containerID="87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6"
Oct 30 20:28:09.182372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.182144    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-688df59c46-jvswg"]
Oct 30 20:28:09.182796 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:09.182771    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6\": container with ID starting with 87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6 not found: ID does not exist" containerID="87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6"
Oct 30 20:28:09.182909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.182806    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6"} err="failed to get container status \"87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6\": rpc error: code = NotFound desc = could not find container \"87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6\": container with ID starting with 87318e8dbd1e67b9fa5faa14d4e86e9ca924abea847f8f4a8894b18c5f75eaf6 not found: ID does not exist"
Oct 30 20:28:09.182909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.182832    4519 scope.go:122] "RemoveContainer" containerID="ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3"
Oct 30 20:28:09.195923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.195900    4519 scope.go:122] "RemoveContainer" containerID="ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3"
Oct 30 20:28:09.197623 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:09.197579    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3\": container with ID starting with ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3 not found: ID does not exist" containerID="ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3"
Oct 30 20:28:09.197824 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.197795    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3"} err="failed to get container status \"ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3\": rpc error: code = NotFound desc = could not find container \"ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3\": container with ID starting with ebbecca04faff444a4a664b2b3218a5817162038a5bd03838c0335767d6c6ad3 not found: ID does not exist"
Oct 30 20:28:09.204517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.204496    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"]
Oct 30 20:28:09.210540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.210513    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-6d966bdc7f-n5gnq"]
Oct 30 20:28:09.613811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.613777    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3298c124-63ac-4ccd-87f2-98002a466cd7" path="/var/lib/kubelet/pods/3298c124-63ac-4ccd-87f2-98002a466cd7/volumes"
Oct 30 20:28:09.614222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.614208    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5fdd18-17f4-4aeb-9ce3-42e134996662" path="/var/lib/kubelet/pods/5e5fdd18-17f4-4aeb-9ce3-42e134996662/volumes"
Oct 30 20:28:09.614617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:09.614603    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" path="/var/lib/kubelet/pods/ddbdb14d-001a-4e7b-a922-2ecbe27f3f30/volumes"
Oct 30 20:28:10.400860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.400753    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:10.522390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.522329    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-openshift-ca\") pod \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") "
Oct 30 20:28:10.522648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.522546    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea19d21-85d0-4d51-88cd-bacde4ebbe19-kube-api-access-pd7h7\" (UniqueName: \"kubernetes.io/projected/bea19d21-85d0-4d51-88cd-bacde4ebbe19-kube-api-access-pd7h7\") pod \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") "
Oct 30 20:28:10.522736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.522676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-xks-ca\") pod \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\" (UID: \"bea19d21-85d0-4d51-88cd-bacde4ebbe19\") "
Oct 30 20:28:10.522838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.522800    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-openshift-ca" pod "bea19d21-85d0-4d51-88cd-bacde4ebbe19" (UID: "bea19d21-85d0-4d51-88cd-bacde4ebbe19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:10.523245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.523217    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:10.525146 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.525108    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea19d21-85d0-4d51-88cd-bacde4ebbe19-kube-api-access-pd7h7" pod "bea19d21-85d0-4d51-88cd-bacde4ebbe19" (UID: "bea19d21-85d0-4d51-88cd-bacde4ebbe19"). InnerVolumeSpecName "kube-api-access-pd7h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:10.528835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.528810    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-xks-ca" pod "bea19d21-85d0-4d51-88cd-bacde4ebbe19" (UID: "bea19d21-85d0-4d51-88cd-bacde4ebbe19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:10.624458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.624420    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea19d21-85d0-4d51-88cd-bacde4ebbe19-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:10.624458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:10.624452    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7h7\" (UniqueName: \"kubernetes.io/projected/bea19d21-85d0-4d51-88cd-bacde4ebbe19-kube-api-access-pd7h7\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:11.146151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:11.146120    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"
Oct 30 20:28:11.146151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:11.146145    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp" event={"ID":"bea19d21-85d0-4d51-88cd-bacde4ebbe19","Type":"ContainerDied","Data":"c34c22afdb9a84a2af95803a7d2ccacaa5cc276b476142bb1210804a8e4b9de2"}
Oct 30 20:28:11.146381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:11.146192    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c34c22afdb9a84a2af95803a7d2ccacaa5cc276b476142bb1210804a8e4b9de2"
Oct 30 20:28:12.032037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:12.031999    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"]
Oct 30 20:28:12.043751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:12.043718    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f956601-jhbrd"]
Oct 30 20:28:13.613000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:13.612959    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfefa741-d9ab-48fb-aad8-e32ef65182c2" path="/var/lib/kubelet/pods/bfefa741-d9ab-48fb-aad8-e32ef65182c2/volumes"
Oct 30 20:28:15.695580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:15.695546    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"]
Oct 30 20:28:15.696025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:15.695779    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy" containerID="cri-o://e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845" gracePeriod=30
Oct 30 20:28:16.030458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.030371    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"]
Oct 30 20:28:16.034240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.034212    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4396420e-sz5ml"]
Oct 30 20:28:16.224223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224187    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"]
Oct 30 20:28:16.224724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224711    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy"
Oct 30 20:28:16.224777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224725    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3085fd-c4f8-431f-a214-6ad1bfd1288e" containerName="istio-proxy"
Oct 30 20:28:16.224777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224742    4519 cpu_manager.go:414] "containerMap: removing container" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy"
Oct 30 20:28:16.224777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224747    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy"
Oct 30 20:28:16.224777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224762    4519 cpu_manager.go:414] "containerMap: removing container" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy"
Oct 30 20:28:16.224777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224767    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy"
Oct 30 20:28:16.224777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224776    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5e5fdd18-17f4-4aeb-9ce3-42e134996662" containerName="payload-pre-processing"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224782    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5fdd18-17f4-4aeb-9ce3-42e134996662" containerName="payload-pre-processing"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224805    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" containerName="payload-processing"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224810    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" containerName="payload-processing"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224821    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bea19d21-85d0-4d51-88cd-bacde4ebbe19" containerName="revoke-keys"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224827    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea19d21-85d0-4d51-88cd-bacde4ebbe19" containerName="revoke-keys"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224906    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224915    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5fdd18-17f4-4aeb-9ce3-42e134996662" containerName="payload-pre-processing"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224926    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbdb14d-001a-4e7b-a922-2ecbe27f3f30" containerName="payload-processing"
Oct 30 20:28:16.224977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.224933    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea19d21-85d0-4d51-88cd-bacde4ebbe19" containerName="revoke-keys"
Oct 30 20:28:16.229630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.229600    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.247382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.247351    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"]
Oct 30 20:28:16.248436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.248412    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:16.248585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.248546    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:28:16.332912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.332818    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.333081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.333043    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trcz5\" (UniqueName: \"kubernetes.io/projected/13793c6b-385e-4cf1-bb84-cfe3dc69557f-kube-api-access-trcz5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.333249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.333230    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.434392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.434355    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trcz5\" (UniqueName: \"kubernetes.io/projected/13793c6b-385e-4cf1-bb84-cfe3dc69557f-kube-api-access-trcz5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.434620 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.434521    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.434620 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.434612    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.435346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.435313    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.437259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.437228    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.457277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.457252    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trcz5\" (UniqueName: \"kubernetes.io/projected/13793c6b-385e-4cf1-bb84-cfe3dc69557f-kube-api-access-trcz5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.545462 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.545425    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:16.886848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:16.886737    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"]
Oct 30 20:28:16.898277 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:28:16.898237    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13793c6b_385e_4cf1_bb84_cfe3dc69557f.slice/crio-e8f50070d551b43c89b1bca991b20d41ce7acea86f29e18957cb8cc0b4adff42 WatchSource:0}: Error finding container e8f50070d551b43c89b1bca991b20d41ce7acea86f29e18957cb8cc0b4adff42: Status 404 returned error can't find the container with id e8f50070d551b43c89b1bca991b20d41ce7acea86f29e18957cb8cc0b4adff42
Oct 30 20:28:17.231042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:17.230996    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg" event={"ID":"13793c6b-385e-4cf1-bb84-cfe3dc69557f","Type":"ContainerStarted","Data":"e8f50070d551b43c89b1bca991b20d41ce7acea86f29e18957cb8cc0b4adff42"}
Oct 30 20:28:17.614192 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:17.614110    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a502c5e-7d1a-4489-9f51-52d525a03b94" path="/var/lib/kubelet/pods/1a502c5e-7d1a-4489-9f51-52d525a03b94/volumes"
Oct 30 20:28:18.242195 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:18.242144    4519 generic.go:356] "Generic (PLEG): container finished" podID="13793c6b-385e-4cf1-bb84-cfe3dc69557f" containerID="c8e77b89a2546f0fde83a27b17dc909dbfa495db702d3ccf862b28bf5323707c" exitCode=0
Oct 30 20:28:18.242607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:18.242217    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg" event={"ID":"13793c6b-385e-4cf1-bb84-cfe3dc69557f","Type":"ContainerDied","Data":"c8e77b89a2546f0fde83a27b17dc909dbfa495db702d3ccf862b28bf5323707c"}
Oct 30 20:28:19.242459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.240569    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:28:19.242459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.240622    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:28:19.428528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.428497    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:19.487395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.487338    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13793c6b-385e-4cf1-bb84-cfe3dc69557f-kube-api-access-trcz5\" (UniqueName: \"kubernetes.io/projected/13793c6b-385e-4cf1-bb84-cfe3dc69557f-kube-api-access-trcz5\") pod \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") "
Oct 30 20:28:19.487664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.487439    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-xks-ca\") pod \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") "
Oct 30 20:28:19.487664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.487624    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-openshift-ca\") pod \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\" (UID: \"13793c6b-385e-4cf1-bb84-cfe3dc69557f\") "
Oct 30 20:28:19.488386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.488352    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-openshift-ca" pod "13793c6b-385e-4cf1-bb84-cfe3dc69557f" (UID: "13793c6b-385e-4cf1-bb84-cfe3dc69557f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:19.490822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.490752    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-xks-ca" pod "13793c6b-385e-4cf1-bb84-cfe3dc69557f" (UID: "13793c6b-385e-4cf1-bb84-cfe3dc69557f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:19.491004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.490976    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13793c6b-385e-4cf1-bb84-cfe3dc69557f-kube-api-access-trcz5" pod "13793c6b-385e-4cf1-bb84-cfe3dc69557f" (UID: "13793c6b-385e-4cf1-bb84-cfe3dc69557f"). InnerVolumeSpecName "kube-api-access-trcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:19.589501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.589424    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trcz5\" (UniqueName: \"kubernetes.io/projected/13793c6b-385e-4cf1-bb84-cfe3dc69557f-kube-api-access-trcz5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:19.589501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.589497    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:19.589694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:19.589516    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13793c6b-385e-4cf1-bb84-cfe3dc69557f-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:20.267548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.267509    4519 generic.go:356] "Generic (PLEG): container finished" podID="29106e39-77cd-4a3c-8766-984760dc28ad" containerID="5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0" exitCode=0
Oct 30 20:28:20.268022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.267581    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" event={"ID":"29106e39-77cd-4a3c-8766-984760dc28ad","Type":"ContainerDied","Data":"5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0"}
Oct 30 20:28:20.268112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.268088    4519 scope.go:122] "RemoveContainer" containerID="5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0"
Oct 30 20:28:20.269538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.269511    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"
Oct 30 20:28:20.269674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.269522    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg" event={"ID":"13793c6b-385e-4cf1-bb84-cfe3dc69557f","Type":"ContainerDied","Data":"e8f50070d551b43c89b1bca991b20d41ce7acea86f29e18957cb8cc0b4adff42"}
Oct 30 20:28:20.269674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.269586    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8f50070d551b43c89b1bca991b20d41ce7acea86f29e18957cb8cc0b4adff42"
Oct 30 20:28:20.270323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.270284    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:28:20.270428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.270383    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:28:20.270518 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:20.270430    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:28:21.036148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.036122    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:28:21.109341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109301    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6155870-2e23-4484-8c1d-1b34e19e2bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6155870-2e23-4484-8c1d-1b34e19e2bca-istiod-ca-cert\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109373    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-certs\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109456    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-data\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109501    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-credential-socket\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109533    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-kube-api-access-zmmt4\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-kube-api-access-zmmt4\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109579    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-token\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-token\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109624    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-envoy\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109698    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-certs" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:21.109823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109785    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-socket\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.109934 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.109919    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-podinfo\") pod \"c6155870-2e23-4484-8c1d-1b34e19e2bca\" (UID: \"c6155870-2e23-4484-8c1d-1b34e19e2bca\") "
Oct 30 20:28:21.110108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.110085    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-data" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:21.110850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.110523    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.110850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.110550    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.111129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.110972    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-socket" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:21.111295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.111254    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-credential-socket" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:21.111770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.111747    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6155870-2e23-4484-8c1d-1b34e19e2bca-istiod-ca-cert" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:21.112739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.112714    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-podinfo" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:28:21.113075 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.113038    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-token" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:21.115846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.115393    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-kube-api-access-zmmt4" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "kube-api-access-zmmt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:21.116086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.116052    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-envoy" pod "c6155870-2e23-4484-8c1d-1b34e19e2bca" (UID: "c6155870-2e23-4484-8c1d-1b34e19e2bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:21.211878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.211836    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6155870-2e23-4484-8c1d-1b34e19e2bca-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.211878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.211878    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.212089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.211896    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmmt4\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-kube-api-access-zmmt4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.212089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.211911    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.212089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.211924    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.212089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.211937    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6155870-2e23-4484-8c1d-1b34e19e2bca-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.212089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.211951    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6155870-2e23-4484-8c1d-1b34e19e2bca-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:21.286510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.286380    4519 generic.go:356] "Generic (PLEG): container finished" podID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerID="e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845" exitCode=0
Oct 30 20:28:21.286510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.286437    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"
Oct 30 20:28:21.286510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.286496    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" event={"ID":"c6155870-2e23-4484-8c1d-1b34e19e2bca","Type":"ContainerDied","Data":"e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845"}
Oct 30 20:28:21.287078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.286594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx" event={"ID":"c6155870-2e23-4484-8c1d-1b34e19e2bca","Type":"ContainerDied","Data":"8df88b10a7c3b194782113c7c8a9c5ca1abbfd14bb644b1d83a4dbac3e733862"}
Oct 30 20:28:21.287078 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.286628    4519 scope.go:122] "RemoveContainer" containerID="e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845"
Oct 30 20:28:21.290704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.290662    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" event={"ID":"29106e39-77cd-4a3c-8766-984760dc28ad","Type":"ContainerStarted","Data":"4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e"}
Oct 30 20:28:21.303520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.303460    4519 scope.go:122] "RemoveContainer" containerID="0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee"
Oct 30 20:28:21.316025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.315997    4519 scope.go:122] "RemoveContainer" containerID="e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845"
Oct 30 20:28:21.316349 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:21.316329    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845\": container with ID starting with e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845 not found: ID does not exist" containerID="e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845"
Oct 30 20:28:21.316448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.316359    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845"} err="failed to get container status \"e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845\": rpc error: code = NotFound desc = could not find container \"e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845\": container with ID starting with e5f43d938291462e4ddb5f6b29e93971bff244175aa3953415d2ee8a8a8e0845 not found: ID does not exist"
Oct 30 20:28:21.316448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.316378    4519 scope.go:122] "RemoveContainer" containerID="0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee"
Oct 30 20:28:21.317069 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:21.317046    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee\": container with ID starting with 0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee not found: ID does not exist" containerID="0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee"
Oct 30 20:28:21.317182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.317074    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee"} err="failed to get container status \"0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee\": rpc error: code = NotFound desc = could not find container \"0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee\": container with ID starting with 0153f5eed27640a96359d2c32cd8251e7e4e798dbbab60f5f972bb4f4fa397ee not found: ID does not exist"
Oct 30 20:28:21.339868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.339839    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"]
Oct 30 20:28:21.364542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.364454    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f35cc1-openshift-default-6446c44dcf-qpnnx"]
Oct 30 20:28:21.618141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:21.618053    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" path="/var/lib/kubelet/pods/c6155870-2e23-4484-8c1d-1b34e19e2bca/volumes"
Oct 30 20:28:22.020903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:22.019774    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:28:22.020903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:22.019879    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 20:28:22.020903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:22.019916    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 20:28:22.032909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:22.032875    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"]
Oct 30 20:28:22.044482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:22.044434    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da205a19-bzbp7"]
Oct 30 20:28:23.019362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:23.019321    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 20:28:23.019801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:23.019386    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 20:28:23.613923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:23.613883    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5455daa-b677-4e82-8663-16c4ccaa3143" path="/var/lib/kubelet/pods/c5455daa-b677-4e82-8663-16c4ccaa3143/volumes"
Oct 30 20:28:24.020328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:24.020285    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 20:28:24.020751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:24.020355    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 20:28:25.019279 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:25.019242    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 20:28:25.019507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:25.019300    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 20:28:26.019387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:26.019349    4519 patch_prober.go:29] interesting pod/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 20:28:26.019848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:26.019436    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 20:28:27.054491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:27.046780    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"]
Oct 30 20:28:27.059907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:27.057722    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:28:27.059907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:27.059120    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:28:27.064617 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:27.064594    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:28:27.068777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:27.068750    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33e7d88-lldgr"]
Oct 30 20:28:27.613906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:27.613867    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e73240e-bbb2-40fa-87c1-eeb47ee496b5" path="/var/lib/kubelet/pods/4e73240e-bbb2-40fa-87c1-eeb47ee496b5/volumes"
Oct 30 20:28:32.222938 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:28:32.222897    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:28:33.037290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:33.037252    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"]
Oct 30 20:28:33.050660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:33.050628    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebaeeeb-sdbxr"]
Oct 30 20:28:33.612985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:33.612947    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4c3c22-0d2f-42a3-ac04-d3e585b57154" path="/var/lib/kubelet/pods/eb4c3c22-0d2f-42a3-ac04-d3e585b57154/volumes"
Oct 30 20:28:37.029803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:37.029764    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"]
Oct 30 20:28:37.041631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:37.041603    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da59d6e3-hkxmw"]
Oct 30 20:28:37.612690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:37.612652    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1daca35-8117-4cd3-83e6-34b2edf18c2f" path="/var/lib/kubelet/pods/b1daca35-8117-4cd3-83e6-34b2edf18c2f/volumes"
Oct 30 20:28:38.751122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:38.751082    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"]
Oct 30 20:28:38.764550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:38.764515    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30e92c-xxhhp"]
Oct 30 20:28:39.614158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:39.614120    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea19d21-85d0-4d51-88cd-bacde4ebbe19" path="/var/lib/kubelet/pods/bea19d21-85d0-4d51-88cd-bacde4ebbe19/volumes"
Oct 30 20:28:44.031380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:44.031330    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"]
Oct 30 20:28:44.035141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:44.035103    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8733451b-r94fz"]
Oct 30 20:28:45.614872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:45.614832    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a6abac-bec4-48fb-a537-14fd3fec31cc" path="/var/lib/kubelet/pods/d9a6abac-bec4-48fb-a537-14fd3fec31cc/volumes"
Oct 30 20:28:46.600196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600157    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"]
Oct 30 20:28:46.600713 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600699    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy"
Oct 30 20:28:46.600774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600715    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy"
Oct 30 20:28:46.600774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600723    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy"
Oct 30 20:28:46.600774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600729    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy"
Oct 30 20:28:46.600774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600750    4519 cpu_manager.go:414] "containerMap: removing container" podUID="13793c6b-385e-4cf1-bb84-cfe3dc69557f" containerName="revoke-keys"
Oct 30 20:28:46.600774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600756    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13793c6b-385e-4cf1-bb84-cfe3dc69557f" containerName="revoke-keys"
Oct 30 20:28:46.600938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600830    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy"
Oct 30 20:28:46.600938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600839    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6155870-2e23-4484-8c1d-1b34e19e2bca" containerName="istio-proxy"
Oct 30 20:28:46.600938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600849    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="13793c6b-385e-4cf1-bb84-cfe3dc69557f" containerName="revoke-keys"
Oct 30 20:28:46.600938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.600857    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="85027075-443e-4323-a372-86b86a6dd098" containerName="istio-proxy"
Oct 30 20:28:46.605868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.605846    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.625986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.625958    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"]
Oct 30 20:28:46.641143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.641083    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:46.641516 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.641497    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:28:46.708639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.708606    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qs5\" (UniqueName: \"kubernetes.io/projected/4525bfac-c74c-4ba9-bc54-1b7349d58c23-kube-api-access-z6qs5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.708841 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.708692    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.708943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.708922    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.810359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.810312    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.810599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.810528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.810682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.810618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qs5\" (UniqueName: \"kubernetes.io/projected/4525bfac-c74c-4ba9-bc54-1b7349d58c23-kube-api-access-z6qs5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.811036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.811013    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.813218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.813195    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.830855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.830822    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qs5\" (UniqueName: \"kubernetes.io/projected/4525bfac-c74c-4ba9-bc54-1b7349d58c23-kube-api-access-z6qs5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:46.920013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:46.919979    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:47.228099 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:28:47.228060    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4525bfac_c74c_4ba9_bc54_1b7349d58c23.slice/crio-60d7ea2d02c9a1f0186823edb00dfa229cb580980eee146eb1a0bb52b6b09e12 WatchSource:0}: Error finding container 60d7ea2d02c9a1f0186823edb00dfa229cb580980eee146eb1a0bb52b6b09e12: Status 404 returned error can't find the container with id 60d7ea2d02c9a1f0186823edb00dfa229cb580980eee146eb1a0bb52b6b09e12
Oct 30 20:28:47.229883 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:47.229864    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"]
Oct 30 20:28:47.604023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:47.603983    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb" event={"ID":"4525bfac-c74c-4ba9-bc54-1b7349d58c23","Type":"ContainerStarted","Data":"60d7ea2d02c9a1f0186823edb00dfa229cb580980eee146eb1a0bb52b6b09e12"}
Oct 30 20:28:48.032301 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:48.032259    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"]
Oct 30 20:28:48.041598 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:48.041568    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3c945-qjtv7"]
Oct 30 20:28:48.613134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:48.613090    4519 generic.go:356] "Generic (PLEG): container finished" podID="4525bfac-c74c-4ba9-bc54-1b7349d58c23" containerID="0657595e4ccd522b20998a96c10b420c96c175928e8e355d2ed72a6ebd6ec0a6" exitCode=0
Oct 30 20:28:48.613321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:48.613165    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb" event={"ID":"4525bfac-c74c-4ba9-bc54-1b7349d58c23","Type":"ContainerDied","Data":"0657595e4ccd522b20998a96c10b420c96c175928e8e355d2ed72a6ebd6ec0a6"}
Oct 30 20:28:49.613566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.613532    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56c44d3-bcc0-4e39-b64d-db292a39e418" path="/var/lib/kubelet/pods/c56c44d3-bcc0-4e39-b64d-db292a39e418/volumes"
Oct 30 20:28:49.813176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.813145    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:49.856272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.856236    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-openshift-ca\") pod \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") "
Oct 30 20:28:49.856448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.856329    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4525bfac-c74c-4ba9-bc54-1b7349d58c23-kube-api-access-z6qs5\" (UniqueName: \"kubernetes.io/projected/4525bfac-c74c-4ba9-bc54-1b7349d58c23-kube-api-access-z6qs5\") pod \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") "
Oct 30 20:28:49.856571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.856490    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-xks-ca\") pod \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\" (UID: \"4525bfac-c74c-4ba9-bc54-1b7349d58c23\") "
Oct 30 20:28:49.857686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.857656    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-openshift-ca" pod "4525bfac-c74c-4ba9-bc54-1b7349d58c23" (UID: "4525bfac-c74c-4ba9-bc54-1b7349d58c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:49.859862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.859833    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-xks-ca" pod "4525bfac-c74c-4ba9-bc54-1b7349d58c23" (UID: "4525bfac-c74c-4ba9-bc54-1b7349d58c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:49.860974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.860948    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4525bfac-c74c-4ba9-bc54-1b7349d58c23-kube-api-access-z6qs5" pod "4525bfac-c74c-4ba9-bc54-1b7349d58c23" (UID: "4525bfac-c74c-4ba9-bc54-1b7349d58c23"). InnerVolumeSpecName "kube-api-access-z6qs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:49.958066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.958034    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:49.958066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.958067    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4525bfac-c74c-4ba9-bc54-1b7349d58c23-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:49.958268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:49.958076    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6qs5\" (UniqueName: \"kubernetes.io/projected/4525bfac-c74c-4ba9-bc54-1b7349d58c23-kube-api-access-z6qs5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:28:50.707264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.707193    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb" event={"ID":"4525bfac-c74c-4ba9-bc54-1b7349d58c23","Type":"ContainerDied","Data":"60d7ea2d02c9a1f0186823edb00dfa229cb580980eee146eb1a0bb52b6b09e12"}
Oct 30 20:28:50.707867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.707281    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60d7ea2d02c9a1f0186823edb00dfa229cb580980eee146eb1a0bb52b6b09e12"
Oct 30 20:28:50.707867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.707651    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"
Oct 30 20:28:50.986194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.986087    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"]
Oct 30 20:28:50.989429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.986971    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4525bfac-c74c-4ba9-bc54-1b7349d58c23" containerName="revoke-keys"
Oct 30 20:28:50.989429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.987000    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4525bfac-c74c-4ba9-bc54-1b7349d58c23" containerName="revoke-keys"
Oct 30 20:28:50.989429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.987194    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4525bfac-c74c-4ba9-bc54-1b7349d58c23" containerName="revoke-keys"
Oct 30 20:28:50.992534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.992306    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"]
Oct 30 20:28:50.992534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:50.992436    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.009127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.007892    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-843ea8-dockercfg-z7qcv\""
Oct 30 20:28:51.009127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.008020    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-843ea8\""
Oct 30 20:28:51.076819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.076777    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f594f2c-af03-4c8d-98c5-4d3ae876650a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.077084 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.076958    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n8j\" (UniqueName: \"kubernetes.io/projected/6f594f2c-af03-4c8d-98c5-4d3ae876650a-kube-api-access-92n8j\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.178225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.178169    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n8j\" (UniqueName: \"kubernetes.io/projected/6f594f2c-af03-4c8d-98c5-4d3ae876650a-kube-api-access-92n8j\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.178448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.178426    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f594f2c-af03-4c8d-98c5-4d3ae876650a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.179155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.179127    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f594f2c-af03-4c8d-98c5-4d3ae876650a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.197335 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.197294    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n8j\" (UniqueName: \"kubernetes.io/projected/6f594f2c-af03-4c8d-98c5-4d3ae876650a-kube-api-access-92n8j\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.312214 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.312131    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:51.639209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.639170    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"]
Oct 30 20:28:51.719457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:51.719415    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9" event={"ID":"6f594f2c-af03-4c8d-98c5-4d3ae876650a","Type":"ContainerStarted","Data":"4d7ed78696191d500a21a15024ac68f60d6c1b1f013c2ab62d2ac94330b7c588"}
Oct 30 20:28:52.728682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:52.728640    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9" event={"ID":"6f594f2c-af03-4c8d-98c5-4d3ae876650a","Type":"ContainerStarted","Data":"ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6"}
Oct 30 20:28:52.729118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:52.728817    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:28:52.770899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:52.770835    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9" podStartSLOduration=2.7708080109999997 podStartE2EDuration="2.770808011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:52.767684133 +0000 UTC m=+3257.268387857" watchObservedRunningTime="2026-10-30 20:28:52.770808011 +0000 UTC m=+3257.271511737"
Oct 30 20:28:53.808650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:53.808612    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"]
Oct 30 20:28:53.818585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:53.818552    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a165e5-nsjvg"]
Oct 30 20:28:55.613070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:28:55.613033    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13793c6b-385e-4cf1-bb84-cfe3dc69557f" path="/var/lib/kubelet/pods/13793c6b-385e-4cf1-bb84-cfe3dc69557f/volumes"
Oct 30 20:29:01.132436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.132396    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"]
Oct 30 20:29:01.133021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.132654    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy" containerID="cri-o://428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c" gracePeriod=30
Oct 30 20:29:01.695265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.695220    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"]
Oct 30 20:29:01.701311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.701286    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.718229 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.718201    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"]
Oct 30 20:29:01.739696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.739668    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:01.740194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.740166    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:29:01.744102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.744076    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.744242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.744111    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.744309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.744255    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zg6c\" (UniqueName: \"kubernetes.io/projected/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-kube-api-access-5zg6c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.846007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.845972    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zg6c\" (UniqueName: \"kubernetes.io/projected/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-kube-api-access-5zg6c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.846201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.846155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.846201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.846175    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.846857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.846833    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.848900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.848869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:01.865199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:01.865173    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zg6c\" (UniqueName: \"kubernetes.io/projected/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-kube-api-access-5zg6c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:02.017649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:02.017563    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:02.225181 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:29:02.225144    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:29:02.327304 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:02.327274    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f05c6b8_806c_4425_b6fc_0df78aa0ac92.slice/crio-48b2a78253228a0cb61e9d8a5f5739a4d8105f87c3364373ed188a10f6f03881 WatchSource:0}: Error finding container 48b2a78253228a0cb61e9d8a5f5739a4d8105f87c3364373ed188a10f6f03881: Status 404 returned error can't find the container with id 48b2a78253228a0cb61e9d8a5f5739a4d8105f87c3364373ed188a10f6f03881
Oct 30 20:29:02.328880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:02.328848    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"]
Oct 30 20:29:02.827305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:02.827270    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt" event={"ID":"2f05c6b8-806c-4425-b6fc-0df78aa0ac92","Type":"ContainerStarted","Data":"2eb22be62f5442deaf7039e2428d12c404fdd10a2429454b3c779c3821fd5d09"}
Oct 30 20:29:02.827502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:02.827325    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt" event={"ID":"2f05c6b8-806c-4425-b6fc-0df78aa0ac92","Type":"ContainerStarted","Data":"48b2a78253228a0cb61e9d8a5f5739a4d8105f87c3364373ed188a10f6f03881"}
Oct 30 20:29:02.878497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:02.875280    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt" podStartSLOduration=1.875255951 podStartE2EDuration="1.875255951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:02.865348424 +0000 UTC m=+3267.366052159" watchObservedRunningTime="2026-10-30 20:29:02.875255951 +0000 UTC m=+3267.375959676"
Oct 30 20:29:03.738572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:03.738537    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:29:03.838767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:03.838729    4519 generic.go:356] "Generic (PLEG): container finished" podID="2f05c6b8-806c-4425-b6fc-0df78aa0ac92" containerID="2eb22be62f5442deaf7039e2428d12c404fdd10a2429454b3c779c3821fd5d09" exitCode=0
Oct 30 20:29:03.838945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:03.838807    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt" event={"ID":"2f05c6b8-806c-4425-b6fc-0df78aa0ac92","Type":"ContainerDied","Data":"2eb22be62f5442deaf7039e2428d12c404fdd10a2429454b3c779c3821fd5d09"}
Oct 30 20:29:05.011245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.011213    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:05.092881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.092838    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-kube-api-access-5zg6c\" (UniqueName: \"kubernetes.io/projected/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-kube-api-access-5zg6c\") pod \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") "
Oct 30 20:29:05.093100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.092939    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-openshift-ca\") pod \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") "
Oct 30 20:29:05.093186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.093149    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-xks-ca\") pod \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\" (UID: \"2f05c6b8-806c-4425-b6fc-0df78aa0ac92\") "
Oct 30 20:29:05.098846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.096391    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-openshift-ca" pod "2f05c6b8-806c-4425-b6fc-0df78aa0ac92" (UID: "2f05c6b8-806c-4425-b6fc-0df78aa0ac92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:05.098846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.096410    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-xks-ca" pod "2f05c6b8-806c-4425-b6fc-0df78aa0ac92" (UID: "2f05c6b8-806c-4425-b6fc-0df78aa0ac92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:05.098846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.096533    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-kube-api-access-5zg6c" pod "2f05c6b8-806c-4425-b6fc-0df78aa0ac92" (UID: "2f05c6b8-806c-4425-b6fc-0df78aa0ac92"). InnerVolumeSpecName "kube-api-access-5zg6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:05.195652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.195615    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:05.195652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.195650    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zg6c\" (UniqueName: \"kubernetes.io/projected/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-kube-api-access-5zg6c\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:05.195846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.195664    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f05c6b8-806c-4425-b6fc-0df78aa0ac92-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:05.858005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.857971    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"
Oct 30 20:29:05.858236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.858001    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt" event={"ID":"2f05c6b8-806c-4425-b6fc-0df78aa0ac92","Type":"ContainerDied","Data":"48b2a78253228a0cb61e9d8a5f5739a4d8105f87c3364373ed188a10f6f03881"}
Oct 30 20:29:05.858236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:05.858060    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b2a78253228a0cb61e9d8a5f5739a4d8105f87c3364373ed188a10f6f03881"
Oct 30 20:29:06.439107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.439080    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 20:29:06.510115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510083    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-podinfo\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-socket\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510201    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-token\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-token\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510241    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-envoy\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510312    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-credential-socket\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510355    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istiod-ca-cert\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510442    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-kube-api-access-9wb8z\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-kube-api-access-9wb8z\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510527    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-data\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510603    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-certs\") pod \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\" (UID: \"e89b6b48-a456-4ed3-b951-bb3d3dd498c1\") "
Oct 30 20:29:06.510829 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510742    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-credential-socket" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:06.510945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.510917    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-socket" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:06.511142 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.511118    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-certs" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:06.511288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.511235    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istiod-ca-cert" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:06.511288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.511282    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.511419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.511303    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.511419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.511319    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.511708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.511657    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-data" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:06.513090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.513045    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-podinfo" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:29:06.513212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.513145    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-envoy" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:06.513212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.513185    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-kube-api-access-9wb8z" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "kube-api-access-9wb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:06.513508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.513454    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-token" pod "e89b6b48-a456-4ed3-b951-bb3d3dd498c1" (UID: "e89b6b48-a456-4ed3-b951-bb3d3dd498c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:06.612423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.612390    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.612423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.612420    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.612423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.612429    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.612684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.612437    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.612684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.612446    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wb8z\" (UniqueName: \"kubernetes.io/projected/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-kube-api-access-9wb8z\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.612684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.612455    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89b6b48-a456-4ed3-b951-bb3d3dd498c1-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:06.869523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.869480    4519 generic.go:356] "Generic (PLEG): container finished" podID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerID="428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c" exitCode=0
Oct 30 20:29:06.869723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.869540    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"
Oct 30 20:29:06.869723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.869571    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" event={"ID":"e89b6b48-a456-4ed3-b951-bb3d3dd498c1","Type":"ContainerDied","Data":"428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c"}
Oct 30 20:29:06.869723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.869631    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf" event={"ID":"e89b6b48-a456-4ed3-b951-bb3d3dd498c1","Type":"ContainerDied","Data":"dd3ab14897d6b7dea05572124514699a7527d18d38f529085f4a2e94b88aaaa6"}
Oct 30 20:29:06.869723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.869660    4519 scope.go:122] "RemoveContainer" containerID="428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c"
Oct 30 20:29:06.881874 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.881853    4519 scope.go:122] "RemoveContainer" containerID="991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204"
Oct 30 20:29:06.897747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.897724    4519 scope.go:122] "RemoveContainer" containerID="428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c"
Oct 30 20:29:06.898030 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:29:06.898010    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c\": container with ID starting with 428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c not found: ID does not exist" containerID="428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c"
Oct 30 20:29:06.898101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.898047    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c"} err="failed to get container status \"428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c\": rpc error: code = NotFound desc = could not find container \"428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c\": container with ID starting with 428f69e46547186a7c3d9b9e734bc5a37d5659bfaf163dcbe95f91accf90905c not found: ID does not exist"
Oct 30 20:29:06.898101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.898063    4519 scope.go:122] "RemoveContainer" containerID="991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204"
Oct 30 20:29:06.898303 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:29:06.898286    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204\": container with ID starting with 991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204 not found: ID does not exist" containerID="991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204"
Oct 30 20:29:06.898365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.898309    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204"} err="failed to get container status \"991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204\": rpc error: code = NotFound desc = could not find container \"991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204\": container with ID starting with 991a819033f654432df15e496e85bf30f275b1d20a03a5c1df8593b042521204 not found: ID does not exist"
Oct 30 20:29:06.925206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.925179    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"]
Oct 30 20:29:06.937217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:06.937191    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a165e5-openshift-default-648688dcd9-mkgpf"]
Oct 30 20:29:07.613533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:07.613498    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" path="/var/lib/kubelet/pods/e89b6b48-a456-4ed3-b951-bb3d3dd498c1/volumes"
Oct 30 20:29:09.045882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.045840    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"]
Oct 30 20:29:09.046634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046616    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy"
Oct 30 20:29:09.046691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046636    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy"
Oct 30 20:29:09.046739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046695    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy"
Oct 30 20:29:09.046739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046703    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy"
Oct 30 20:29:09.046739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046714    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2f05c6b8-806c-4425-b6fc-0df78aa0ac92" containerName="revoke-keys"
Oct 30 20:29:09.046739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046723    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f05c6b8-806c-4425-b6fc-0df78aa0ac92" containerName="revoke-keys"
Oct 30 20:29:09.046887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046840    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy"
Oct 30 20:29:09.046887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046860    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89b6b48-a456-4ed3-b951-bb3d3dd498c1" containerName="istio-proxy"
Oct 30 20:29:09.046887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.046872    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f05c6b8-806c-4425-b6fc-0df78aa0ac92" containerName="revoke-keys"
Oct 30 20:29:09.052286 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.052264    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.072817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.072786    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"]
Oct 30 20:29:09.080583 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.080553    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:09.080791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.080687    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:29:09.144404 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.144365    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.144621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.144409    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4k9\" (UniqueName: \"kubernetes.io/projected/c35d0c3c-998f-4df2-ad41-59726cf97762-kube-api-access-9t4k9\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.144621 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.144561    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.246294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.246256    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.246532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.246409    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.246532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.246442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4k9\" (UniqueName: \"kubernetes.io/projected/c35d0c3c-998f-4df2-ad41-59726cf97762-kube-api-access-9t4k9\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.247291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.247259    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.248879 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.248857    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.266421 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.266398    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4k9\" (UniqueName: \"kubernetes.io/projected/c35d0c3c-998f-4df2-ad41-59726cf97762-kube-api-access-9t4k9\") pod \"maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.369790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.369704    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:09.681769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.681733    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"]
Oct 30 20:29:09.682003 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:09.681770    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35d0c3c_998f_4df2_ad41_59726cf97762.slice/crio-cde968b09d762883230da75d338dcd1ac893f1b974fea58343d306daeecec980 WatchSource:0}: Error finding container cde968b09d762883230da75d338dcd1ac893f1b974fea58343d306daeecec980: Status 404 returned error can't find the container with id cde968b09d762883230da75d338dcd1ac893f1b974fea58343d306daeecec980
Oct 30 20:29:09.902390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:09.902348    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv" event={"ID":"c35d0c3c-998f-4df2-ad41-59726cf97762","Type":"ContainerStarted","Data":"cde968b09d762883230da75d338dcd1ac893f1b974fea58343d306daeecec980"}
Oct 30 20:29:10.913460 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:10.913421    4519 generic.go:356] "Generic (PLEG): container finished" podID="c35d0c3c-998f-4df2-ad41-59726cf97762" containerID="8d89fffc778a034c1dc8d6c4b627e098bc07aa4aff183b92cfd5265822caeab2" exitCode=0
Oct 30 20:29:10.913875 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:10.913497    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv" event={"ID":"c35d0c3c-998f-4df2-ad41-59726cf97762","Type":"ContainerDied","Data":"8d89fffc778a034c1dc8d6c4b627e098bc07aa4aff183b92cfd5265822caeab2"}
Oct 30 20:29:11.211988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:11.211878    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"]
Oct 30 20:29:11.215498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:11.215456    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c8fa69-n64gb"]
Oct 30 20:29:11.613559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:11.613461    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4525bfac-c74c-4ba9-bc54-1b7349d58c23" path="/var/lib/kubelet/pods/4525bfac-c74c-4ba9-bc54-1b7349d58c23/volumes"
Oct 30 20:29:12.084769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.084737    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:12.205890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.205799    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-xks-ca\") pod \"c35d0c3c-998f-4df2-ad41-59726cf97762\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") "
Oct 30 20:29:12.205890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.205856    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35d0c3c-998f-4df2-ad41-59726cf97762-kube-api-access-9t4k9\" (UniqueName: \"kubernetes.io/projected/c35d0c3c-998f-4df2-ad41-59726cf97762-kube-api-access-9t4k9\") pod \"c35d0c3c-998f-4df2-ad41-59726cf97762\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") "
Oct 30 20:29:12.206129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.206000    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-openshift-ca\") pod \"c35d0c3c-998f-4df2-ad41-59726cf97762\" (UID: \"c35d0c3c-998f-4df2-ad41-59726cf97762\") "
Oct 30 20:29:12.206388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.206356    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-openshift-ca" pod "c35d0c3c-998f-4df2-ad41-59726cf97762" (UID: "c35d0c3c-998f-4df2-ad41-59726cf97762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:12.206669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.206645    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:12.208526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.208499    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35d0c3c-998f-4df2-ad41-59726cf97762-kube-api-access-9t4k9" pod "c35d0c3c-998f-4df2-ad41-59726cf97762" (UID: "c35d0c3c-998f-4df2-ad41-59726cf97762"). InnerVolumeSpecName "kube-api-access-9t4k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:12.208696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.208669    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-xks-ca" pod "c35d0c3c-998f-4df2-ad41-59726cf97762" (UID: "c35d0c3c-998f-4df2-ad41-59726cf97762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:12.307789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.307750    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35d0c3c-998f-4df2-ad41-59726cf97762-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:12.307789 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.307784    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4k9\" (UniqueName: \"kubernetes.io/projected/c35d0c3c-998f-4df2-ad41-59726cf97762-kube-api-access-9t4k9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:12.930813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.930782    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"
Oct 30 20:29:12.931018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.930823    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv" event={"ID":"c35d0c3c-998f-4df2-ad41-59726cf97762","Type":"ContainerDied","Data":"cde968b09d762883230da75d338dcd1ac893f1b974fea58343d306daeecec980"}
Oct 30 20:29:12.931018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:12.930881    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cde968b09d762883230da75d338dcd1ac893f1b974fea58343d306daeecec980"
Oct 30 20:29:18.459508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:18.459439    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"]
Oct 30 20:29:18.460026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:18.459782    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy" containerID="cri-o://271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056" gracePeriod=30
Oct 30 20:29:19.016824 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.016782    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"]
Oct 30 20:29:19.017353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.017339    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c35d0c3c-998f-4df2-ad41-59726cf97762" containerName="revoke-keys"
Oct 30 20:29:19.017401 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.017356    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35d0c3c-998f-4df2-ad41-59726cf97762" containerName="revoke-keys"
Oct 30 20:29:19.017498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.017480    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35d0c3c-998f-4df2-ad41-59726cf97762" containerName="revoke-keys"
Oct 30 20:29:19.020641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.020621    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.041206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.041181    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"]
Oct 30 20:29:19.056802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.056774    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:29:19.057065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.057043    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:19.102577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.102549    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.102720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.102609    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzzk\" (UniqueName: \"kubernetes.io/projected/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-kube-api-access-mmzzk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.102831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.102807    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.204344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.204300    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.204567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.204376    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.204567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.204433    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzzk\" (UniqueName: \"kubernetes.io/projected/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-kube-api-access-mmzzk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.205134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.205095    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.206957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.206936    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.224684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.224659    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzzk\" (UniqueName: \"kubernetes.io/projected/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-kube-api-access-mmzzk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.335347 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.335267    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:19.656801 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:19.656767    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0312bbcf_86d4_4ef7_a39e_8e0cf8ea5a6e.slice/crio-b3a9e2a24ccd318b6fb424006a93a66a60005fb98328d1365dff7a3c5876de9c WatchSource:0}: Error finding container b3a9e2a24ccd318b6fb424006a93a66a60005fb98328d1365dff7a3c5876de9c: Status 404 returned error can't find the container with id b3a9e2a24ccd318b6fb424006a93a66a60005fb98328d1365dff7a3c5876de9c
Oct 30 20:29:19.657845 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.657817    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"]
Oct 30 20:29:19.992290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.992253    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv" event={"ID":"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e","Type":"ContainerStarted","Data":"370920a3c90651a861f58cc9999f52a3feb5b51674e804d61047e3a3730c977d"}
Oct 30 20:29:19.992503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:19.992310    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv" event={"ID":"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e","Type":"ContainerStarted","Data":"b3a9e2a24ccd318b6fb424006a93a66a60005fb98328d1365dff7a3c5876de9c"}
Oct 30 20:29:20.037159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:20.037104    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv" podStartSLOduration=2.037091062 podStartE2EDuration="2.037091062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:20.033066131 +0000 UTC m=+3284.533769854" watchObservedRunningTime="2026-10-30 20:29:20.037091062 +0000 UTC m=+3284.537794785"
Oct 30 20:29:21.002023 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:21.001979    4519 generic.go:356] "Generic (PLEG): container finished" podID="0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" containerID="370920a3c90651a861f58cc9999f52a3feb5b51674e804d61047e3a3730c977d" exitCode=0
Oct 30 20:29:21.002433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:21.002060    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv" event={"ID":"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e","Type":"ContainerDied","Data":"370920a3c90651a861f58cc9999f52a3feb5b51674e804d61047e3a3730c977d"}
Oct 30 20:29:22.211393 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.211366    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:22.345440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.345347    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-xks-ca\") pod \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") "
Oct 30 20:29:22.345440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.345395    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-openshift-ca\") pod \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") "
Oct 30 20:29:22.345730 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.345674    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-kube-api-access-mmzzk\" (UniqueName: \"kubernetes.io/projected/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-kube-api-access-mmzzk\") pod \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\" (UID: \"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e\") "
Oct 30 20:29:22.345850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.345823    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-openshift-ca" pod "0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" (UID: "0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:22.346134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.346115    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:22.347732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.347703    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-xks-ca" pod "0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" (UID: "0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:22.348191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.348166    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-kube-api-access-mmzzk" pod "0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" (UID: "0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e"). InnerVolumeSpecName "kube-api-access-mmzzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:22.447446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.447402    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmzzk\" (UniqueName: \"kubernetes.io/projected/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-kube-api-access-mmzzk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:22.447446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:22.447445    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.018929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.018895    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv" event={"ID":"0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e","Type":"ContainerDied","Data":"b3a9e2a24ccd318b6fb424006a93a66a60005fb98328d1365dff7a3c5876de9c"}
Oct 30 20:29:23.019117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.018935    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"
Oct 30 20:29:23.019117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.018945    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3a9e2a24ccd318b6fb424006a93a66a60005fb98328d1365dff7a3c5876de9c"
Oct 30 20:29:23.734739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.734679    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 20:29:23.865246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865202    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccb4412e-9702-4277-bb1c-3f658f977837-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb4412e-9702-4277-bb1c-3f658f977837-istio-podinfo\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865302    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-certs\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865382    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-envoy\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865439    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-kube-api-access-ql8hh\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-kube-api-access-ql8hh\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865488    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-istio-token\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-istio-token\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-credential-socket\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865639    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-data\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865678    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb4412e-9702-4277-bb1c-3f658f977837-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb4412e-9702-4277-bb1c-3f658f977837-istiod-ca-cert\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.865849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865685    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-certs" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:23.865849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865758    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-socket\") pod \"ccb4412e-9702-4277-bb1c-3f658f977837\" (UID: \"ccb4412e-9702-4277-bb1c-3f658f977837\") "
Oct 30 20:29:23.866017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.865912    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-credential-socket" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:23.866077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.866002    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-data" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:23.866250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.866223    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-socket" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:23.866449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.866241    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb4412e-9702-4277-bb1c-3f658f977837-istiod-ca-cert" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:23.866581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.866389    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.866705 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.866687    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.866832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.866797    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.868103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.868068    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-kube-api-access-ql8hh" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "kube-api-access-ql8hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:23.868230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.868183    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-istio-token" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:23.868332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.868309    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-envoy" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:23.868572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.868547    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccb4412e-9702-4277-bb1c-3f658f977837-istio-podinfo" pod "ccb4412e-9702-4277-bb1c-3f658f977837" (UID: "ccb4412e-9702-4277-bb1c-3f658f977837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:29:23.968340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.968291    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.968340 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.968337    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccb4412e-9702-4277-bb1c-3f658f977837-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.968575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.968351    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccb4412e-9702-4277-bb1c-3f658f977837-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.968575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.968365    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8hh\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-kube-api-access-ql8hh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.968575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.968381    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccb4412e-9702-4277-bb1c-3f658f977837-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:23.968575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:23.968395    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccb4412e-9702-4277-bb1c-3f658f977837-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:24.028801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.028712    4519 generic.go:356] "Generic (PLEG): container finished" podID="ccb4412e-9702-4277-bb1c-3f658f977837" containerID="271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056" exitCode=0
Oct 30 20:29:24.028801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.028763    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"
Oct 30 20:29:24.029007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.028790    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" event={"ID":"ccb4412e-9702-4277-bb1c-3f658f977837","Type":"ContainerDied","Data":"271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056"}
Oct 30 20:29:24.029007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.028847    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw" event={"ID":"ccb4412e-9702-4277-bb1c-3f658f977837","Type":"ContainerDied","Data":"0796c5930844092fcfacad76ab25b19a8a97ecf8e0c364f84e8a952463c6d626"}
Oct 30 20:29:24.029007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.028870    4519 scope.go:122] "RemoveContainer" containerID="271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056"
Oct 30 20:29:24.041177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.041158    4519 scope.go:122] "RemoveContainer" containerID="817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f"
Oct 30 20:29:24.054796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.054776    4519 scope.go:122] "RemoveContainer" containerID="271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056"
Oct 30 20:29:24.055122 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:29:24.055092    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056\": container with ID starting with 271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056 not found: ID does not exist" containerID="271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056"
Oct 30 20:29:24.055189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.055135    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056"} err="failed to get container status \"271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056\": rpc error: code = NotFound desc = could not find container \"271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056\": container with ID starting with 271b9748685c83ba02c953da1ae0949c8dce9080917e1f8a079d80578bd37056 not found: ID does not exist"
Oct 30 20:29:24.055189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.055158    4519 scope.go:122] "RemoveContainer" containerID="817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f"
Oct 30 20:29:24.055525 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:29:24.055499    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f\": container with ID starting with 817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f not found: ID does not exist" containerID="817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f"
Oct 30 20:29:24.055630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.055559    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f"} err="failed to get container status \"817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f\": rpc error: code = NotFound desc = could not find container \"817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f\": container with ID starting with 817e7dee8100dae3f173340e08aa822ba39b58c1790ee2a982a5891ba8941d9f not found: ID does not exist"
Oct 30 20:29:24.083740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.083709    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"]
Oct 30 20:29:24.089668 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:24.089644    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c8fa69-openshift-default-7d4cc6788c-bnpgw"]
Oct 30 20:29:25.612480 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:25.612434    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" path="/var/lib/kubelet/pods/ccb4412e-9702-4277-bb1c-3f658f977837/volumes"
Oct 30 20:29:31.972794 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:31.972748    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"]
Oct 30 20:29:31.984359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:31.984316    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87ff9f-fnmkt"]
Oct 30 20:29:32.223669 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:29:32.223572    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:29:33.613731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:33.613694    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f05c6b8-806c-4425-b6fc-0df78aa0ac92" path="/var/lib/kubelet/pods/2f05c6b8-806c-4425-b6fc-0df78aa0ac92/volumes"
Oct 30 20:29:37.434641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:37.432200    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"]
Oct 30 20:29:37.434641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:37.432528    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z" podUID="6571859c-a145-44d4-bbb0-e32ed236cef0" containerName="payload-pre-processing" containerID="cri-o://63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0" gracePeriod=30
Oct 30 20:29:37.843040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:37.843004    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:29:37.913141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:37.913103    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6571859c-a145-44d4-bbb0-e32ed236cef0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6571859c-a145-44d4-bbb0-e32ed236cef0-plugins-config-volume\") pod \"6571859c-a145-44d4-bbb0-e32ed236cef0\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") "
Oct 30 20:29:37.913341 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:37.913166    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6571859c-a145-44d4-bbb0-e32ed236cef0-kube-api-access-7qs9n\" (UniqueName: \"kubernetes.io/projected/6571859c-a145-44d4-bbb0-e32ed236cef0-kube-api-access-7qs9n\") pod \"6571859c-a145-44d4-bbb0-e32ed236cef0\" (UID: \"6571859c-a145-44d4-bbb0-e32ed236cef0\") "
Oct 30 20:29:37.914013 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:37.913981    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6571859c-a145-44d4-bbb0-e32ed236cef0-plugins-config-volume" pod "6571859c-a145-44d4-bbb0-e32ed236cef0" (UID: "6571859c-a145-44d4-bbb0-e32ed236cef0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:37.919996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:37.919954    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6571859c-a145-44d4-bbb0-e32ed236cef0-kube-api-access-7qs9n" pod "6571859c-a145-44d4-bbb0-e32ed236cef0" (UID: "6571859c-a145-44d4-bbb0-e32ed236cef0"). InnerVolumeSpecName "kube-api-access-7qs9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:38.015522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.015441    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6571859c-a145-44d4-bbb0-e32ed236cef0-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:38.015522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.015498    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qs9n\" (UniqueName: \"kubernetes.io/projected/6571859c-a145-44d4-bbb0-e32ed236cef0-kube-api-access-7qs9n\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:38.138321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.138284    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"]
Oct 30 20:29:38.142517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.142486    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c19563-nvrvv"]
Oct 30 20:29:38.178897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.178855    4519 generic.go:356] "Generic (PLEG): container finished" podID="6571859c-a145-44d4-bbb0-e32ed236cef0" containerID="63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0" exitCode=0
Oct 30 20:29:38.179112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.178902    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"
Oct 30 20:29:38.179112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.178933    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z" event={"ID":"6571859c-a145-44d4-bbb0-e32ed236cef0","Type":"ContainerDied","Data":"63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0"}
Oct 30 20:29:38.179112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.178996    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z" event={"ID":"6571859c-a145-44d4-bbb0-e32ed236cef0","Type":"ContainerDied","Data":"faf2b4f25b0ada3ad7856a1592fda8377206487c79c52bff1805644e1724f2af"}
Oct 30 20:29:38.179112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.179030    4519 scope.go:122] "RemoveContainer" containerID="63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0"
Oct 30 20:29:38.192535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.192508    4519 scope.go:122] "RemoveContainer" containerID="63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0"
Oct 30 20:29:38.192833 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:29:38.192805    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0\": container with ID starting with 63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0 not found: ID does not exist" containerID="63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0"
Oct 30 20:29:38.192922 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.192845    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0"} err="failed to get container status \"63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0\": rpc error: code = NotFound desc = could not find container \"63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0\": container with ID starting with 63a863caed0a6432a0950c85eae56675ba2aa2e17d13d3074409817fca7922a0 not found: ID does not exist"
Oct 30 20:29:38.241990 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.241193    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"]
Oct 30 20:29:38.247502 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:38.247391    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-c4f7484c5-m8v6z"]
Oct 30 20:29:39.612844 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:39.612811    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6571859c-a145-44d4-bbb0-e32ed236cef0" path="/var/lib/kubelet/pods/6571859c-a145-44d4-bbb0-e32ed236cef0/volumes"
Oct 30 20:29:39.613250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:39.613234    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35d0c3c-998f-4df2-ad41-59726cf97762" path="/var/lib/kubelet/pods/c35d0c3c-998f-4df2-ad41-59726cf97762/volumes"
Oct 30 20:29:39.714497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:39.714439    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"]
Oct 30 20:29:39.714796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:39.714749    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" containerID="cri-o://06c2f09d4ba76cd63af3fd4fffd477d41555448bf24658218a5b472230ebc62f" gracePeriod=30
Oct 30 20:29:40.483744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.483704    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"]
Oct 30 20:29:40.484264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484248    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" containerName="revoke-keys"
Oct 30 20:29:40.484323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484265    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" containerName="revoke-keys"
Oct 30 20:29:40.484323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484280    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6571859c-a145-44d4-bbb0-e32ed236cef0" containerName="payload-pre-processing"
Oct 30 20:29:40.484323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484286    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6571859c-a145-44d4-bbb0-e32ed236cef0" containerName="payload-pre-processing"
Oct 30 20:29:40.484323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484299    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy"
Oct 30 20:29:40.484323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484304    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy"
Oct 30 20:29:40.484527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484394    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" containerName="revoke-keys"
Oct 30 20:29:40.484527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484404    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6571859c-a145-44d4-bbb0-e32ed236cef0" containerName="payload-pre-processing"
Oct 30 20:29:40.484527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484413    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy"
Oct 30 20:29:40.484527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.484421    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy"
Oct 30 20:29:40.487905 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.487880    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.510689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.510655    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"]
Oct 30 20:29:40.548881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.548847    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.549028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.548900    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.549131 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.549115    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22qc\" (UniqueName: \"kubernetes.io/projected/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-kube-api-access-k22qc\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.561809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.561781    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:40.561809 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.561801    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:29:40.650852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.650808    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22qc\" (UniqueName: \"kubernetes.io/projected/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-kube-api-access-k22qc\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.651334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.650895    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.651334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.650928    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.651672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.651643    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.653590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.653557    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.670632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.670604    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22qc\" (UniqueName: \"kubernetes.io/projected/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-kube-api-access-k22qc\") pod \"maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:40.802252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:40.802166    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:41.003868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:41.003821    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:29:41.004047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:41.003902    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:29:41.125532 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:41.125445    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb2bf3b_af66_4bce_98c3_77906d3d8bf4.slice/crio-f9ac07e7131e2b49455150031b8442885a4a5dcfc7ebe2c638cf83d97bc1dc06 WatchSource:0}: Error finding container f9ac07e7131e2b49455150031b8442885a4a5dcfc7ebe2c638cf83d97bc1dc06: Status 404 returned error can't find the container with id f9ac07e7131e2b49455150031b8442885a4a5dcfc7ebe2c638cf83d97bc1dc06
Oct 30 20:29:41.126183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:41.126159    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"]
Oct 30 20:29:41.208996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:41.208958    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5" event={"ID":"abb2bf3b-af66-4bce-98c3-77906d3d8bf4","Type":"ContainerStarted","Data":"f9ac07e7131e2b49455150031b8442885a4a5dcfc7ebe2c638cf83d97bc1dc06"}
Oct 30 20:29:42.219542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:42.219505    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5" event={"ID":"abb2bf3b-af66-4bce-98c3-77906d3d8bf4","Type":"ContainerStarted","Data":"3f3228b54c505967aa9a9165862e47dfad40d2a8bd800286c7903d2e2779d4b2"}
Oct 30 20:29:42.269808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:42.269757    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5" podStartSLOduration=2.2697428139999998 podStartE2EDuration="2.269742814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:42.267833482 +0000 UTC m=+3306.768537206" watchObservedRunningTime="2026-10-30 20:29:42.269742814 +0000 UTC m=+3306.770446537"
Oct 30 20:29:43.229947 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:43.229910    4519 generic.go:356] "Generic (PLEG): container finished" podID="abb2bf3b-af66-4bce-98c3-77906d3d8bf4" containerID="3f3228b54c505967aa9a9165862e47dfad40d2a8bd800286c7903d2e2779d4b2" exitCode=0
Oct 30 20:29:43.230370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:43.229991    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5" event={"ID":"abb2bf3b-af66-4bce-98c3-77906d3d8bf4","Type":"ContainerDied","Data":"3f3228b54c505967aa9a9165862e47dfad40d2a8bd800286c7903d2e2779d4b2"}
Oct 30 20:29:44.562899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.562866    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:44.658565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.658529    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-xks-ca\") pod \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") "
Oct 30 20:29:44.658775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.658632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-kube-api-access-k22qc\" (UniqueName: \"kubernetes.io/projected/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-kube-api-access-k22qc\") pod \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") "
Oct 30 20:29:44.658775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.658724    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-openshift-ca\") pod \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\" (UID: \"abb2bf3b-af66-4bce-98c3-77906d3d8bf4\") "
Oct 30 20:29:44.659154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.659124    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-openshift-ca" pod "abb2bf3b-af66-4bce-98c3-77906d3d8bf4" (UID: "abb2bf3b-af66-4bce-98c3-77906d3d8bf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:44.659288 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.659233    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:44.661012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.660966    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-xks-ca" pod "abb2bf3b-af66-4bce-98c3-77906d3d8bf4" (UID: "abb2bf3b-af66-4bce-98c3-77906d3d8bf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:44.661218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.661192    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-kube-api-access-k22qc" pod "abb2bf3b-af66-4bce-98c3-77906d3d8bf4" (UID: "abb2bf3b-af66-4bce-98c3-77906d3d8bf4"). InnerVolumeSpecName "kube-api-access-k22qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:44.685741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.685701    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"]
Oct 30 20:29:44.704504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.704058    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-21c4ab-pwrdv"]
Oct 30 20:29:44.763517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.760205    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:44.763517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.760265    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22qc\" (UniqueName: \"kubernetes.io/projected/abb2bf3b-af66-4bce-98c3-77906d3d8bf4-kube-api-access-k22qc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:44.976190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:44.976162    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 20:29:45.063103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istiod-ca-cert\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.063342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-socket\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.063342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063168    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-token\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-token\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.063342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063207    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-data\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.063342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063236    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-certs\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.064058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063364    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-envoy\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.064058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063461    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-podinfo\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.064058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063518    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-kube-api-access-kxljp\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-kube-api-access-kxljp\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.064058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063600    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-credential-socket\") pod \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\" (UID: \"d79bd922-ebb1-48e9-a300-9e0efb1b69d3\") "
Oct 30 20:29:45.064278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063594    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-data" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:45.064278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.063961    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-socket" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:45.064278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.064184    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-credential-socket" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:45.064688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.064265    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-certs" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:45.064746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.064695    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istiod-ca-cert" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:45.066430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.066389    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-envoy" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:45.066600 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.066437    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-podinfo" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:29:45.067447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.067418    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-token" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:45.067659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.067632    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-kube-api-access-kxljp" pod "d79bd922-ebb1-48e9-a300-9e0efb1b69d3" (UID: "d79bd922-ebb1-48e9-a300-9e0efb1b69d3"). InnerVolumeSpecName "kube-api-access-kxljp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:45.165540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165503    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165532    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxljp\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-kube-api-access-kxljp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165545    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165555    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165566    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165579    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165588    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165596    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.165791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.165604    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79bd922-ebb1-48e9-a300-9e0efb1b69d3-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:45.178092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.178043    4519 scope.go:122] "RemoveContainer" containerID="305848aec4ede14d0790c35b7b26082e2eb3fd7891255b267a6ccd2f69c81be1"
Oct 30 20:29:45.189595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.189529    4519 scope.go:122] "RemoveContainer" containerID="89becff8257e45b6f6b6684b265d34cfa0e38e54359a5951918a6a5241b71895"
Oct 30 20:29:45.199071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.199051    4519 scope.go:122] "RemoveContainer" containerID="a5212b395dd59dc1607dac435235de9815e5129ecedcaa7e9c202e660c3ce5d3"
Oct 30 20:29:45.208949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.208928    4519 scope.go:122] "RemoveContainer" containerID="06c2f09d4ba76cd63af3fd4fffd477d41555448bf24658218a5b472230ebc62f"
Oct 30 20:29:45.219011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.218988    4519 scope.go:122] "RemoveContainer" containerID="8ee54d1f0f054104c2c312ec4a5aceaee49b1c733ff789aa88670d6f9e6066ce"
Oct 30 20:29:45.228290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.228270    4519 scope.go:122] "RemoveContainer" containerID="f63df2993c5e459a268cad947b934dcc05de5a039c340e2665696d649a92f722"
Oct 30 20:29:45.237253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.237234    4519 scope.go:122] "RemoveContainer" containerID="14374090c85cba763e7a06d8aa9efca2b2e58092d8152c601c9d0feb17c25779"
Oct 30 20:29:45.246565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.246544    4519 scope.go:122] "RemoveContainer" containerID="707a36453b8e476288bc63b4eab09416bb859bd376d039dc6790bb74fbe256e7"
Oct 30 20:29:45.256725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.256700    4519 scope.go:122] "RemoveContainer" containerID="ad64e03f618835568bfbb440b8c1cc217fa2de6fcc139e7424f921f877fda3ae"
Oct 30 20:29:45.265438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.265411    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"]
Oct 30 20:29:45.298161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.298124    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"]
Oct 30 20:29:45.299979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.299954    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"
Oct 30 20:29:45.300113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.300004    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" event={"ID":"d79bd922-ebb1-48e9-a300-9e0efb1b69d3","Type":"ContainerDied","Data":"06c2f09d4ba76cd63af3fd4fffd477d41555448bf24658218a5b472230ebc62f"}
Oct 30 20:29:45.300113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.300066    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m" event={"ID":"d79bd922-ebb1-48e9-a300-9e0efb1b69d3","Type":"ContainerDied","Data":"28616ed13723808366bcb36565691ebe99fdc585ca1b4a308e6ec7accee103ed"}
Oct 30 20:29:45.304511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.303595    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"
Oct 30 20:29:45.304511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.303615    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5" event={"ID":"abb2bf3b-af66-4bce-98c3-77906d3d8bf4","Type":"ContainerDied","Data":"f9ac07e7131e2b49455150031b8442885a4a5dcfc7ebe2c638cf83d97bc1dc06"}
Oct 30 20:29:45.304511 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.303668    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ac07e7131e2b49455150031b8442885a4a5dcfc7ebe2c638cf83d97bc1dc06"
Oct 30 20:29:45.389267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.389236    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"]
Oct 30 20:29:45.408319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.408272    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-87ff9f-openshift-default-596d595c7c-dqs9m"]
Oct 30 20:29:45.613587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.613553    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e" path="/var/lib/kubelet/pods/0312bbcf-86d4-4ef7-a39e-8e0cf8ea5a6e/volumes"
Oct 30 20:29:45.614086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:45.614071    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" path="/var/lib/kubelet/pods/d79bd922-ebb1-48e9-a300-9e0efb1b69d3/volumes"
Oct 30 20:29:47.032439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.032404    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"]
Oct 30 20:29:47.032955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.032938    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy"
Oct 30 20:29:47.033006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.032956    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy"
Oct 30 20:29:47.033006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.032969    4519 cpu_manager.go:414] "containerMap: removing container" podUID="abb2bf3b-af66-4bce-98c3-77906d3d8bf4" containerName="revoke-keys"
Oct 30 20:29:47.033006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.032975    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb2bf3b-af66-4bce-98c3-77906d3d8bf4" containerName="revoke-keys"
Oct 30 20:29:47.033006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.032985    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy"
Oct 30 20:29:47.033006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.032991    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb4412e-9702-4277-bb1c-3f658f977837" containerName="istio-proxy"
Oct 30 20:29:47.033169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.033077    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy"
Oct 30 20:29:47.033169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.033091    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb2bf3b-af66-4bce-98c3-77906d3d8bf4" containerName="revoke-keys"
Oct 30 20:29:47.038721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.038697    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.072039 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.071278    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"]
Oct 30 20:29:47.073073 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.073040    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:29:47.074011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.073379    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:47.190533 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.190497    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.190739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.190555    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.190739 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.190721    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z22t\" (UniqueName: \"kubernetes.io/projected/fe5432e4-3ab4-4f60-95a2-0322a5326808-kube-api-access-7z22t\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.292332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.292233    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z22t\" (UniqueName: \"kubernetes.io/projected/fe5432e4-3ab4-4f60-95a2-0322a5326808-kube-api-access-7z22t\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.292558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.292397    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.292558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.292442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.293247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.293216    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.294902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.294874    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.320259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.320233    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z22t\" (UniqueName: \"kubernetes.io/projected/fe5432e4-3ab4-4f60-95a2-0322a5326808-kube-api-access-7z22t\") pod \"maas-api-revoke-sub-8a631379-scgc2\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.380106 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.380071    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:47.383194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.383164    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"]
Oct 30 20:29:47.383987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.383966    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy"
Oct 30 20:29:47.384060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.383988    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy"
Oct 30 20:29:47.384122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.384106    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79bd922-ebb1-48e9-a300-9e0efb1b69d3" containerName="istio-proxy"
Oct 30 20:29:47.388085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.388063    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.411196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.411160    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"]
Oct 30 20:29:47.494907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.494861    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.495523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.494964    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvljr\" (UniqueName: \"kubernetes.io/projected/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-kube-api-access-tvljr\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.495523 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.495149    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.596979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.596881    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvljr\" (UniqueName: \"kubernetes.io/projected/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-kube-api-access-tvljr\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.597143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.597048    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.597236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.597187    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.598011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.597984    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.599507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.599461    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.621044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.621020    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvljr\" (UniqueName: \"kubernetes.io/projected/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-kube-api-access-tvljr\") pod \"maas-api-revoke-sub-f0cdc971-8g2mp\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:47.710330 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:47.710301    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5432e4_3ab4_4f60_95a2_0322a5326808.slice/crio-8f2fcefc16136651abc8565b9c26e8f23498b7fedd564d8c0d925618dadfee64 WatchSource:0}: Error finding container 8f2fcefc16136651abc8565b9c26e8f23498b7fedd564d8c0d925618dadfee64: Status 404 returned error can't find the container with id 8f2fcefc16136651abc8565b9c26e8f23498b7fedd564d8c0d925618dadfee64
Oct 30 20:29:47.710566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.710544    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"]
Oct 30 20:29:47.734923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:47.734899    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:48.062801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.062764    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"]
Oct 30 20:29:48.063149 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:48.062809    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6b985c_5bd9_4bff_8c0b_b4921fbecd8e.slice/crio-e178bc0c7f328937133b7bc1a498de7308544745c4b2b713d22e8f2d08048be6 WatchSource:0}: Error finding container e178bc0c7f328937133b7bc1a498de7308544745c4b2b713d22e8f2d08048be6: Status 404 returned error can't find the container with id e178bc0c7f328937133b7bc1a498de7308544745c4b2b713d22e8f2d08048be6
Oct 30 20:29:48.187849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.187804    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"]
Oct 30 20:29:48.188454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.188097    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy" containerID="cri-o://e8ee6ae805c71bdfc28d3bc10472db1c88fba43900538c8401e8e37fb66072ca" gracePeriod=30
Oct 30 20:29:48.374037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.373939    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp" event={"ID":"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e","Type":"ContainerStarted","Data":"e178bc0c7f328937133b7bc1a498de7308544745c4b2b713d22e8f2d08048be6"}
Oct 30 20:29:48.376371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.376328    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2" event={"ID":"fe5432e4-3ab4-4f60-95a2-0322a5326808","Type":"ContainerStarted","Data":"a668e14e8e321d66d87055881fa024d406ca73d2a44692d058e31406fa5c5286"}
Oct 30 20:29:48.376525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.376383    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2" event={"ID":"fe5432e4-3ab4-4f60-95a2-0322a5326808","Type":"ContainerStarted","Data":"8f2fcefc16136651abc8565b9c26e8f23498b7fedd564d8c0d925618dadfee64"}
Oct 30 20:29:48.430496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.430426    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2" podStartSLOduration=2.430412702 podStartE2EDuration="2.430412702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:48.426426546 +0000 UTC m=+3312.927130271" watchObservedRunningTime="2026-10-30 20:29:48.430412702 +0000 UTC m=+3312.931116426"
Oct 30 20:29:48.820755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.820718    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"]
Oct 30 20:29:48.824764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.824744    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:48.854170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.854134    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"]
Oct 30 20:29:48.925803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.925741    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:48.926021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.925919    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8n5\" (UniqueName: \"kubernetes.io/projected/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-kube-api-access-2h8n5\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:48.926097 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:48.926029    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.027458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.027414    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.027642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.027528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8n5\" (UniqueName: \"kubernetes.io/projected/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-kube-api-access-2h8n5\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.027642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.027588    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.028289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.028261    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.030116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.030084    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.051633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.051602    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8n5\" (UniqueName: \"kubernetes.io/projected/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-kube-api-access-2h8n5\") pod \"maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.139059 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.139019    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:49.387501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.387373    4519 generic.go:356] "Generic (PLEG): container finished" podID="ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" containerID="f85526df8905da4cd589e888aedc9947f9a3ec1e372805b80b0f9d8ae564a070" exitCode=0
Oct 30 20:29:49.387501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.387454    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp" event={"ID":"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e","Type":"ContainerDied","Data":"f85526df8905da4cd589e888aedc9947f9a3ec1e372805b80b0f9d8ae564a070"}
Oct 30 20:29:49.389618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.389587    4519 generic.go:356] "Generic (PLEG): container finished" podID="fe5432e4-3ab4-4f60-95a2-0322a5326808" containerID="a668e14e8e321d66d87055881fa024d406ca73d2a44692d058e31406fa5c5286" exitCode=0
Oct 30 20:29:49.389776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.389674    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2" event={"ID":"fe5432e4-3ab4-4f60-95a2-0322a5326808","Type":"ContainerDied","Data":"a668e14e8e321d66d87055881fa024d406ca73d2a44692d058e31406fa5c5286"}
Oct 30 20:29:49.475196 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:49.475144    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc93ebb_ad6d_4cae_8d00_036fce3e529b.slice/crio-c1ade7b216ecf5a18828ed5f3d88fdcfbb486af8cba9d19d9a5f3de02713c7f7 WatchSource:0}: Error finding container c1ade7b216ecf5a18828ed5f3d88fdcfbb486af8cba9d19d9a5f3de02713c7f7: Status 404 returned error can't find the container with id c1ade7b216ecf5a18828ed5f3d88fdcfbb486af8cba9d19d9a5f3de02713c7f7
Oct 30 20:29:49.476911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:49.476882    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"]
Oct 30 20:29:50.401133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.401087    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q" event={"ID":"9bc93ebb-ad6d-4cae-8d00-036fce3e529b","Type":"ContainerStarted","Data":"b8ae50d62e3bbf57a954ba5774cfd35c53833b24bac8bd48ecb7fcc7986b8b4c"}
Oct 30 20:29:50.401772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.401162    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q" event={"ID":"9bc93ebb-ad6d-4cae-8d00-036fce3e529b","Type":"ContainerStarted","Data":"c1ade7b216ecf5a18828ed5f3d88fdcfbb486af8cba9d19d9a5f3de02713c7f7"}
Oct 30 20:29:50.453577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.453514    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q" podStartSLOduration=2.453493597 podStartE2EDuration="2.453493597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:50.449703029 +0000 UTC m=+3314.950406754" watchObservedRunningTime="2026-10-30 20:29:50.453493597 +0000 UTC m=+3314.954197319"
Oct 30 20:29:50.658036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.657967    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:50.661906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.661878    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:50.751379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.751342    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-xks-ca\") pod \"fe5432e4-3ab4-4f60-95a2-0322a5326808\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") "
Oct 30 20:29:50.751646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.751459    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-kube-api-access-tvljr\" (UniqueName: \"kubernetes.io/projected/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-kube-api-access-tvljr\") pod \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") "
Oct 30 20:29:50.751646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.751507    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-openshift-ca\") pod \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") "
Oct 30 20:29:50.751646 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.751569    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5432e4-3ab4-4f60-95a2-0322a5326808-kube-api-access-7z22t\" (UniqueName: \"kubernetes.io/projected/fe5432e4-3ab4-4f60-95a2-0322a5326808-kube-api-access-7z22t\") pod \"fe5432e4-3ab4-4f60-95a2-0322a5326808\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") "
Oct 30 20:29:50.752276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.751685    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-xks-ca\") pod \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\" (UID: \"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e\") "
Oct 30 20:29:50.752276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.751855    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-openshift-ca\") pod \"fe5432e4-3ab4-4f60-95a2-0322a5326808\" (UID: \"fe5432e4-3ab4-4f60-95a2-0322a5326808\") "
Oct 30 20:29:50.752276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.751920    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-openshift-ca" pod "ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" (UID: "ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:50.752725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.752700    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:50.753079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.753050    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-openshift-ca" pod "fe5432e4-3ab4-4f60-95a2-0322a5326808" (UID: "fe5432e4-3ab4-4f60-95a2-0322a5326808"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:50.756554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.756503    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5432e4-3ab4-4f60-95a2-0322a5326808-kube-api-access-7z22t" pod "fe5432e4-3ab4-4f60-95a2-0322a5326808" (UID: "fe5432e4-3ab4-4f60-95a2-0322a5326808"). InnerVolumeSpecName "kube-api-access-7z22t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:50.756943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.756913    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-xks-ca" pod "fe5432e4-3ab4-4f60-95a2-0322a5326808" (UID: "fe5432e4-3ab4-4f60-95a2-0322a5326808"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:50.758970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.758947    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-kube-api-access-tvljr" pod "ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" (UID: "ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e"). InnerVolumeSpecName "kube-api-access-tvljr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:50.773150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.772028    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-xks-ca" pod "ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" (UID: "ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:50.854047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.854002    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvljr\" (UniqueName: \"kubernetes.io/projected/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-kube-api-access-tvljr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:50.854047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.854044    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z22t\" (UniqueName: \"kubernetes.io/projected/fe5432e4-3ab4-4f60-95a2-0322a5326808-kube-api-access-7z22t\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:50.854266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.854059    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:50.854266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.854073    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:50.854266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:50.854087    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5432e4-3ab4-4f60-95a2-0322a5326808-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:51.410790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.410746    4519 generic.go:356] "Generic (PLEG): container finished" podID="9bc93ebb-ad6d-4cae-8d00-036fce3e529b" containerID="b8ae50d62e3bbf57a954ba5774cfd35c53833b24bac8bd48ecb7fcc7986b8b4c" exitCode=0
Oct 30 20:29:51.411303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.410835    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q" event={"ID":"9bc93ebb-ad6d-4cae-8d00-036fce3e529b","Type":"ContainerDied","Data":"b8ae50d62e3bbf57a954ba5774cfd35c53833b24bac8bd48ecb7fcc7986b8b4c"}
Oct 30 20:29:51.412344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.412325    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"
Oct 30 20:29:51.412454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.412358    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2" event={"ID":"fe5432e4-3ab4-4f60-95a2-0322a5326808","Type":"ContainerDied","Data":"8f2fcefc16136651abc8565b9c26e8f23498b7fedd564d8c0d925618dadfee64"}
Oct 30 20:29:51.412454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.412401    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2fcefc16136651abc8565b9c26e8f23498b7fedd564d8c0d925618dadfee64"
Oct 30 20:29:51.414143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.414120    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp" event={"ID":"ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e","Type":"ContainerDied","Data":"e178bc0c7f328937133b7bc1a498de7308544745c4b2b713d22e8f2d08048be6"}
Oct 30 20:29:51.414261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.414156    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"
Oct 30 20:29:51.414261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:51.414157    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e178bc0c7f328937133b7bc1a498de7308544745c4b2b713d22e8f2d08048be6"
Oct 30 20:29:52.582328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.582295    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:52.678834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.678733    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-kube-api-access-2h8n5\" (UniqueName: \"kubernetes.io/projected/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-kube-api-access-2h8n5\") pod \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") "
Oct 30 20:29:52.679040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.678973    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-xks-ca\") pod \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") "
Oct 30 20:29:52.679111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.679067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-openshift-ca\") pod \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\" (UID: \"9bc93ebb-ad6d-4cae-8d00-036fce3e529b\") "
Oct 30 20:29:52.681331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.681300    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-xks-ca" pod "9bc93ebb-ad6d-4cae-8d00-036fce3e529b" (UID: "9bc93ebb-ad6d-4cae-8d00-036fce3e529b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:52.682155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.682126    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-openshift-ca" pod "9bc93ebb-ad6d-4cae-8d00-036fce3e529b" (UID: "9bc93ebb-ad6d-4cae-8d00-036fce3e529b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:52.682629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.682598    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-kube-api-access-2h8n5" pod "9bc93ebb-ad6d-4cae-8d00-036fce3e529b" (UID: "9bc93ebb-ad6d-4cae-8d00-036fce3e529b"). InnerVolumeSpecName "kube-api-access-2h8n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:52.780732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.780607    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h8n5\" (UniqueName: \"kubernetes.io/projected/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-kube-api-access-2h8n5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:52.780732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.780653    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:52.780732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:52.780669    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc93ebb-ad6d-4cae-8d00-036fce3e529b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.433688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.433621    4519 generic.go:356] "Generic (PLEG): container finished" podID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerID="e8ee6ae805c71bdfc28d3bc10472db1c88fba43900538c8401e8e37fb66072ca" exitCode=0
Oct 30 20:29:53.433688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.433669    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" event={"ID":"4b38681b-3ca5-4c6b-8e5a-ff36c5569665","Type":"ContainerDied","Data":"e8ee6ae805c71bdfc28d3bc10472db1c88fba43900538c8401e8e37fb66072ca"}
Oct 30 20:29:53.433972 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.433756    4519 scope.go:122] "RemoveContainer" containerID="1feaf89c42b04dbba2bd1d28a3ecadc86642385d7dc8384c3cd6258ff10c601d"
Oct 30 20:29:53.437821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.437784    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q" event={"ID":"9bc93ebb-ad6d-4cae-8d00-036fce3e529b","Type":"ContainerDied","Data":"c1ade7b216ecf5a18828ed5f3d88fdcfbb486af8cba9d19d9a5f3de02713c7f7"}
Oct 30 20:29:53.437984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.437849    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ade7b216ecf5a18828ed5f3d88fdcfbb486af8cba9d19d9a5f3de02713c7f7"
Oct 30 20:29:53.437984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.437925    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"
Oct 30 20:29:53.462402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.462336    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:29:53.515177 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515141    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"]
Oct 30 20:29:53.515790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515763    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy"
Oct 30 20:29:53.515790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515787    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515811    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fe5432e4-3ab4-4f60-95a2-0322a5326808" containerName="revoke-keys"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515819    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5432e4-3ab4-4f60-95a2-0322a5326808" containerName="revoke-keys"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515846    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515854    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515867    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9bc93ebb-ad6d-4cae-8d00-036fce3e529b" containerName="revoke-keys"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515875    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc93ebb-ad6d-4cae-8d00-036fce3e529b" containerName="revoke-keys"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515887    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" containerName="revoke-keys"
Oct 30 20:29:53.515968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.515896    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" containerName="revoke-keys"
Oct 30 20:29:53.516230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.516010    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy"
Oct 30 20:29:53.516230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.516019    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5432e4-3ab4-4f60-95a2-0322a5326808" containerName="revoke-keys"
Oct 30 20:29:53.516230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.516029    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" containerName="istio-proxy"
Oct 30 20:29:53.516230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.516035    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc93ebb-ad6d-4cae-8d00-036fce3e529b" containerName="revoke-keys"
Oct 30 20:29:53.516230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.516042    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" containerName="revoke-keys"
Oct 30 20:29:53.519597 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.519571    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.543331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.543289    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"]
Oct 30 20:29:53.543792 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.543763    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:53.544808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.544783    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.592785    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istiod-ca-cert\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.592887    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-kube-api-access-wghfv\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-kube-api-access-wghfv\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593008    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-podinfo\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593042    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-envoy\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593081    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-certs\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593178    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-socket\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-data\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593248    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istiod-ca-cert" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593287    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-credential-socket\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593332    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-token\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-token\") pod \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\" (UID: \"4b38681b-3ca5-4c6b-8e5a-ff36c5569665\") "
Oct 30 20:29:53.594209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.593901    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.595588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.594952    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-certs" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:53.595588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.595015    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-data" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:53.595588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.595063    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-credential-socket" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:53.595588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.595311    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-socket" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:53.597956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.597927    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-kube-api-access-wghfv" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "kube-api-access-wghfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:53.598066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.597967    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-token" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:53.599012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.598985    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-envoy" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:53.599274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.599252    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-podinfo" pod "4b38681b-3ca5-4c6b-8e5a-ff36c5569665" (UID: "4b38681b-3ca5-4c6b-8e5a-ff36c5569665"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:29:53.695182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695132    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54fs\" (UniqueName: \"kubernetes.io/projected/3afb7643-aa97-4840-9780-690d3a3538fd-kube-api-access-m54fs\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.695363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695195    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.695518 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695491    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.695704 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695685    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.695751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695708    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.695751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695719    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.695751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695729    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.695751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695738    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.695751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695749    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.695910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695760    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.695910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.695771    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wghfv\" (UniqueName: \"kubernetes.io/projected/4b38681b-3ca5-4c6b-8e5a-ff36c5569665-kube-api-access-wghfv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:53.796967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.796876    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.796967 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.796962    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54fs\" (UniqueName: \"kubernetes.io/projected/3afb7643-aa97-4840-9780-690d3a3538fd-kube-api-access-m54fs\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.797183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.796984    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.797567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.797542    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.799493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.799433    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.816121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.816092    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54fs\" (UniqueName: \"kubernetes.io/projected/3afb7643-aa97-4840-9780-690d3a3538fd-kube-api-access-m54fs\") pod \"maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:53.835964 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:53.835922    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:54.150525 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:54.150451    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afb7643_aa97_4840_9780_690d3a3538fd.slice/crio-3068f899e78b4dee94a4eedf4e67220670ad3ebfd4c34e3adce0d78f3fbca15e WatchSource:0}: Error finding container 3068f899e78b4dee94a4eedf4e67220670ad3ebfd4c34e3adce0d78f3fbca15e: Status 404 returned error can't find the container with id 3068f899e78b4dee94a4eedf4e67220670ad3ebfd4c34e3adce0d78f3fbca15e
Oct 30 20:29:54.150737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:54.150706    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"]
Oct 30 20:29:54.447034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:54.446993    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42" event={"ID":"3afb7643-aa97-4840-9780-690d3a3538fd","Type":"ContainerStarted","Data":"3068f899e78b4dee94a4eedf4e67220670ad3ebfd4c34e3adce0d78f3fbca15e"}
Oct 30 20:29:54.449003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:54.448974    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"
Oct 30 20:29:54.449185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:54.448975    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg" event={"ID":"4b38681b-3ca5-4c6b-8e5a-ff36c5569665","Type":"ContainerDied","Data":"01db642a5b6713950f275508d4e44f4bcd378c82e8bc5a67626c822e39ed13da"}
Oct 30 20:29:54.449185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:54.449114    4519 scope.go:122] "RemoveContainer" containerID="e8ee6ae805c71bdfc28d3bc10472db1c88fba43900538c8401e8e37fb66072ca"
Oct 30 20:29:54.507346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:54.507309    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"]
Oct 30 20:29:54.518801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:54.518769    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c19563-openshift-default-6df46cbddf-h8ncg"]
Oct 30 20:29:55.460222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:55.460179    4519 generic.go:356] "Generic (PLEG): container finished" podID="3afb7643-aa97-4840-9780-690d3a3538fd" containerID="18fbd58e91af3c35036cb8b43e9f7ce868cdf858b5e781cff1ef4aab2f67fa5b" exitCode=0
Oct 30 20:29:55.460691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:55.460235    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42" event={"ID":"3afb7643-aa97-4840-9780-690d3a3538fd","Type":"ContainerDied","Data":"18fbd58e91af3c35036cb8b43e9f7ce868cdf858b5e781cff1ef4aab2f67fa5b"}
Oct 30 20:29:55.614038 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:55.614006    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b38681b-3ca5-4c6b-8e5a-ff36c5569665" path="/var/lib/kubelet/pods/4b38681b-3ca5-4c6b-8e5a-ff36c5569665/volumes"
Oct 30 20:29:56.625135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.625105    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:56.731613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.731526    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-xks-ca\") pod \"3afb7643-aa97-4840-9780-690d3a3538fd\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") "
Oct 30 20:29:56.731613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.731577    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afb7643-aa97-4840-9780-690d3a3538fd-kube-api-access-m54fs\" (UniqueName: \"kubernetes.io/projected/3afb7643-aa97-4840-9780-690d3a3538fd-kube-api-access-m54fs\") pod \"3afb7643-aa97-4840-9780-690d3a3538fd\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") "
Oct 30 20:29:56.731850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.731724    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-openshift-ca\") pod \"3afb7643-aa97-4840-9780-690d3a3538fd\" (UID: \"3afb7643-aa97-4840-9780-690d3a3538fd\") "
Oct 30 20:29:56.732434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.732388    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-openshift-ca" pod "3afb7643-aa97-4840-9780-690d3a3538fd" (UID: "3afb7643-aa97-4840-9780-690d3a3538fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:56.734117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.734089    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afb7643-aa97-4840-9780-690d3a3538fd-kube-api-access-m54fs" pod "3afb7643-aa97-4840-9780-690d3a3538fd" (UID: "3afb7643-aa97-4840-9780-690d3a3538fd"). InnerVolumeSpecName "kube-api-access-m54fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:56.734438 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.734411    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-xks-ca" pod "3afb7643-aa97-4840-9780-690d3a3538fd" (UID: "3afb7643-aa97-4840-9780-690d3a3538fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:56.833951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.833915    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:56.833951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.833946    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb7643-aa97-4840-9780-690d3a3538fd-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:56.833951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:56.833957    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m54fs\" (UniqueName: \"kubernetes.io/projected/3afb7643-aa97-4840-9780-690d3a3538fd-kube-api-access-m54fs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:29:57.360639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.360595    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"]
Oct 30 20:29:57.361383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.361358    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3afb7643-aa97-4840-9780-690d3a3538fd" containerName="revoke-keys"
Oct 30 20:29:57.361487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.361386    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afb7643-aa97-4840-9780-690d3a3538fd" containerName="revoke-keys"
Oct 30 20:29:57.361590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.361576    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afb7643-aa97-4840-9780-690d3a3538fd" containerName="revoke-keys"
Oct 30 20:29:57.365757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.365735    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.388194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.388160    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"]
Oct 30 20:29:57.491442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.491400    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42" event={"ID":"3afb7643-aa97-4840-9780-690d3a3538fd","Type":"ContainerDied","Data":"3068f899e78b4dee94a4eedf4e67220670ad3ebfd4c34e3adce0d78f3fbca15e"}
Oct 30 20:29:57.491658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.491474    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"
Oct 30 20:29:57.491658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.491515    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3068f899e78b4dee94a4eedf4e67220670ad3ebfd4c34e3adce0d78f3fbca15e"
Oct 30 20:29:57.542905 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.542871    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.543074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.543031    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fmhk\" (UniqueName: \"kubernetes.io/projected/02e58824-5528-42b1-9d5c-3852f6876cd5-kube-api-access-6fmhk\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.543136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.543089    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.644847 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.644764    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.645311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.644895    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fmhk\" (UniqueName: \"kubernetes.io/projected/02e58824-5528-42b1-9d5c-3852f6876cd5-kube-api-access-6fmhk\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.645311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.644956    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.645519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.645496    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.647310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.647288    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.671344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.671298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fmhk\" (UniqueName: \"kubernetes.io/projected/02e58824-5528-42b1-9d5c-3852f6876cd5-kube-api-access-6fmhk\") pod \"maas-api-revoke-sub-3aab56ab-b8h7m\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.684948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.684915    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:29:57.744514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.740696    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"]
Oct 30 20:29:57.746547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.746518    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.750395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.750363    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"]
Oct 30 20:29:57.847303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.847265    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.847530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.847325    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78jz\" (UniqueName: \"kubernetes.io/projected/a5024961-e814-4fb2-953a-38f1920d7a08-kube-api-access-z78jz\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.847605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.847540    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.948668 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.948635    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.948831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.948720    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.948831 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.948767    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78jz\" (UniqueName: \"kubernetes.io/projected/a5024961-e814-4fb2-953a-38f1920d7a08-kube-api-access-z78jz\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.949482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.949437    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.951226 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.951202    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:57.969087 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:57.969064    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78jz\" (UniqueName: \"kubernetes.io/projected/a5024961-e814-4fb2-953a-38f1920d7a08-kube-api-access-z78jz\") pod \"maas-api-revoke-sub-04a93742-lrffp\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:58.006339 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:58.006297    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e58824_5528_42b1_9d5c_3852f6876cd5.slice/crio-a79fd0049352995f8a60fa8881f907894aaccf46d86c13487734a4b410164dc9 WatchSource:0}: Error finding container a79fd0049352995f8a60fa8881f907894aaccf46d86c13487734a4b410164dc9: Status 404 returned error can't find the container with id a79fd0049352995f8a60fa8881f907894aaccf46d86c13487734a4b410164dc9
Oct 30 20:29:58.007030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.007005    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"]
Oct 30 20:29:58.079357 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.079322    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:29:58.252017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.251934    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"]
Oct 30 20:29:58.396988 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:29:58.396956    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5024961_e814_4fb2_953a_38f1920d7a08.slice/crio-e25a63d39989adcf7ea8059add402c5e53d067e0e0f7a32f5c5d6b619fb9f62c WatchSource:0}: Error finding container e25a63d39989adcf7ea8059add402c5e53d067e0e0f7a32f5c5d6b619fb9f62c: Status 404 returned error can't find the container with id e25a63d39989adcf7ea8059add402c5e53d067e0e0f7a32f5c5d6b619fb9f62c
Oct 30 20:29:58.398345 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.398320    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"]
Oct 30 20:29:58.499792 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.499752    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp" event={"ID":"a5024961-e814-4fb2-953a-38f1920d7a08","Type":"ContainerStarted","Data":"e25a63d39989adcf7ea8059add402c5e53d067e0e0f7a32f5c5d6b619fb9f62c"}
Oct 30 20:29:58.501819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.501789    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m" event={"ID":"02e58824-5528-42b1-9d5c-3852f6876cd5","Type":"ContainerStarted","Data":"087e170fc8b91c8da1bbce03497c6732685e549305bfeed1f216fa20a7c7a49c"}
Oct 30 20:29:58.501985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.501849    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m" event={"ID":"02e58824-5528-42b1-9d5c-3852f6876cd5","Type":"ContainerStarted","Data":"a79fd0049352995f8a60fa8881f907894aaccf46d86c13487734a4b410164dc9"}
Oct 30 20:29:58.548702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:58.548601    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m" podStartSLOduration=1.548585849 podStartE2EDuration="1.548585849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:58.544642154 +0000 UTC m=+3323.045345879" watchObservedRunningTime="2026-10-30 20:29:58.548585849 +0000 UTC m=+3323.049289595"
Oct 30 20:29:59.511283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:59.511248    4519 generic.go:356] "Generic (PLEG): container finished" podID="02e58824-5528-42b1-9d5c-3852f6876cd5" containerID="087e170fc8b91c8da1bbce03497c6732685e549305bfeed1f216fa20a7c7a49c" exitCode=0
Oct 30 20:29:59.511801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:59.511360    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m" event={"ID":"02e58824-5528-42b1-9d5c-3852f6876cd5","Type":"ContainerDied","Data":"087e170fc8b91c8da1bbce03497c6732685e549305bfeed1f216fa20a7c7a49c"}
Oct 30 20:29:59.513538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:59.513515    4519 generic.go:356] "Generic (PLEG): container finished" podID="a5024961-e814-4fb2-953a-38f1920d7a08" containerID="2aa5233c6cf5cb2f2fd62b3f641dacae49bc35bdedba4f654510616546ee0897" exitCode=0
Oct 30 20:29:59.513674 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:29:59.513544    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp" event={"ID":"a5024961-e814-4fb2-953a-38f1920d7a08","Type":"ContainerDied","Data":"2aa5233c6cf5cb2f2fd62b3f641dacae49bc35bdedba4f654510616546ee0897"}
Oct 30 20:30:00.159432 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.159390    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"]
Oct 30 20:30:00.163698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.163674    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.175979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.175951    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"]
Oct 30 20:30:00.179321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.179292    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcab0c-a105-4dd1-a417-f324ba5bf305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.179504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.179474    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njdn\" (UniqueName: \"kubernetes.io/projected/bcbcab0c-a105-4dd1-a417-f324ba5bf305-kube-api-access-4njdn\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.179761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.179743    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"]
Oct 30 20:30:00.179862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.179845    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.183654 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.183627    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"]
Oct 30 20:30:00.183772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.183739    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.187547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.187526    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.205929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.203694    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"]
Oct 30 20:30:00.228907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.228872    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"]
Oct 30 20:30:00.235571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.233453    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.237551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.237386    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"]
Oct 30 20:30:00.244598 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.244558    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"]
Oct 30 20:30:00.253594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.252539    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"]
Oct 30 20:30:00.272701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.272665    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"]
Oct 30 20:30:00.272823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.272777    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.273174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.273154    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"]
Oct 30 20:30:00.277422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.277391    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.279105 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.279082    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"]
Oct 30 20:30:00.283700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283572    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95t2p\" (UniqueName: \"kubernetes.io/projected/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-kube-api-access-95t2p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.283785 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283717    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.283785 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283775    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7dv\" (UniqueName: \"kubernetes.io/projected/42663d1a-4550-41bb-920b-52fdeb79efee-kube-api-access-7k7dv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.283913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283835    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.283913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283874    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcab0c-a105-4dd1-a417-f324ba5bf305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.284021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283921    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffd813f-fc60-482f-8268-f32c32262e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.284021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283966    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbs8b\" (UniqueName: \"kubernetes.io/projected/02c434ed-d6fa-4857-9642-f0627666212e-kube-api-access-dbs8b\") pod \"maas-api-key-cleanup-29889870-b5rg2\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.284021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.283999    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx9n6\" (UniqueName: \"kubernetes.io/projected/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-kube-api-access-cx9n6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.284186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.284070    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c434ed-d6fa-4857-9642-f0627666212e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-b5rg2\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.284186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.284100    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfzf\" (UniqueName: \"kubernetes.io/projected/5ffd813f-fc60-482f-8268-f32c32262e8d-kube-api-access-kvfzf\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.284186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.284139    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42663d1a-4550-41bb-920b-52fdeb79efee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.284355 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.284186    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njdn\" (UniqueName: \"kubernetes.io/projected/bcbcab0c-a105-4dd1-a417-f324ba5bf305-kube-api-access-4njdn\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.285319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.285291    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcab0c-a105-4dd1-a417-f324ba5bf305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.287562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.287537    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"]
Oct 30 20:30:00.293662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.293636    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main" containerID="cri-o://4f92532538612876c8b604a2fcfb8a8c134ea860ea7b543e4f985ff5e679ca6f" gracePeriod=45
Oct 30 20:30:00.298262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.298234    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"]
Oct 30 20:30:00.319866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.319834    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njdn\" (UniqueName: \"kubernetes.io/projected/bcbcab0c-a105-4dd1-a417-f324ba5bf305-kube-api-access-4njdn\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.329542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.325703    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main" containerID="cri-o://9590bfa4a5c5602002f0a77ff8a461d61ea9c5fac43c0e011ac945f2b838b77b" gracePeriod=45
Oct 30 20:30:00.385538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385501    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.385696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385572    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffd813f-fc60-482f-8268-f32c32262e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.385696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385604    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbs8b\" (UniqueName: \"kubernetes.io/projected/02c434ed-d6fa-4857-9642-f0627666212e-kube-api-access-dbs8b\") pod \"maas-api-key-cleanup-29889870-b5rg2\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.385696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385619    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx9n6\" (UniqueName: \"kubernetes.io/projected/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-kube-api-access-cx9n6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.385696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385688    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c434ed-d6fa-4857-9642-f0627666212e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-b5rg2\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.385945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385718    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfzf\" (UniqueName: \"kubernetes.io/projected/5ffd813f-fc60-482f-8268-f32c32262e8d-kube-api-access-kvfzf\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.385945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385754    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42663d1a-4550-41bb-920b-52fdeb79efee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.385945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385824    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ba0576-c136-4bff-ba0d-925da9b0488e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.385945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385852    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95t2p\" (UniqueName: \"kubernetes.io/projected/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-kube-api-access-95t2p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.385945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.385908    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxsl\" (UniqueName: \"kubernetes.io/projected/13ba0576-c136-4bff-ba0d-925da9b0488e-kube-api-access-mgxsl\") pod \"maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.386232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.386021    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.386232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.386081    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7dv\" (UniqueName: \"kubernetes.io/projected/42663d1a-4550-41bb-920b-52fdeb79efee-kube-api-access-7k7dv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.386354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.386317    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.386411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.386367    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffd813f-fc60-482f-8268-f32c32262e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.386592 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.386567    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c434ed-d6fa-4857-9642-f0627666212e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-b5rg2\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.386913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.386891    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.386989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.386933    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42663d1a-4550-41bb-920b-52fdeb79efee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.412306 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.412241    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfzf\" (UniqueName: \"kubernetes.io/projected/5ffd813f-fc60-482f-8268-f32c32262e8d-kube-api-access-kvfzf\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.413104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.413070    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbs8b\" (UniqueName: \"kubernetes.io/projected/02c434ed-d6fa-4857-9642-f0627666212e-kube-api-access-dbs8b\") pod \"maas-api-key-cleanup-29889870-b5rg2\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.413865 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.413839    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95t2p\" (UniqueName: \"kubernetes.io/projected/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-kube-api-access-95t2p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.414161 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.414143    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7dv\" (UniqueName: \"kubernetes.io/projected/42663d1a-4550-41bb-920b-52fdeb79efee-kube-api-access-7k7dv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.414832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.414809    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx9n6\" (UniqueName: \"kubernetes.io/projected/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-kube-api-access-cx9n6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.479512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.478270    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:00.487323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.487237    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ba0576-c136-4bff-ba0d-925da9b0488e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.487323 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.487301    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxsl\" (UniqueName: \"kubernetes.io/projected/13ba0576-c136-4bff-ba0d-925da9b0488e-kube-api-access-mgxsl\") pod \"maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.488344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.488317    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ba0576-c136-4bff-ba0d-925da9b0488e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.499731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.499699    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:00.518354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.518221    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:00.522148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.521025    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxsl\" (UniqueName: \"kubernetes.io/projected/13ba0576-c136-4bff-ba0d-925da9b0488e-kube-api-access-mgxsl\") pod \"maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.532390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.532355    4519 generic.go:356] "Generic (PLEG): container finished" podID="a25a5898-dfb6-4257-935d-e4290aa91888" containerID="9590bfa4a5c5602002f0a77ff8a461d61ea9c5fac43c0e011ac945f2b838b77b" exitCode=0
Oct 30 20:30:00.532552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.532507    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" event={"ID":"a25a5898-dfb6-4257-935d-e4290aa91888","Type":"ContainerDied","Data":"9590bfa4a5c5602002f0a77ff8a461d61ea9c5fac43c0e011ac945f2b838b77b"}
Oct 30 20:30:00.535007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.534482    4519 generic.go:356] "Generic (PLEG): container finished" podID="714c801b-8384-450a-bef2-53760f5efd2a" containerID="4f92532538612876c8b604a2fcfb8a8c134ea860ea7b543e4f985ff5e679ca6f" exitCode=0
Oct 30 20:30:00.535007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.534821    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" event={"ID":"714c801b-8384-450a-bef2-53760f5efd2a","Type":"ContainerDied","Data":"4f92532538612876c8b604a2fcfb8a8c134ea860ea7b543e4f985ff5e679ca6f"}
Oct 30 20:30:00.536916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.536894    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:00.573827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.572706    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:00.599988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.599714    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:00.619866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.619805    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:00.661927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.661549    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.698381    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714c801b-8384-450a-bef2-53760f5efd2a-kube-api-access-qklfh\" (UniqueName: \"kubernetes.io/projected/714c801b-8384-450a-bef2-53760f5efd2a-kube-api-access-qklfh\") pod \"714c801b-8384-450a-bef2-53760f5efd2a\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") "
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.698518    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-model-cache\") pod \"714c801b-8384-450a-bef2-53760f5efd2a\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") "
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.698579    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714c801b-8384-450a-bef2-53760f5efd2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/714c801b-8384-450a-bef2-53760f5efd2a-tls-certs\") pod \"714c801b-8384-450a-bef2-53760f5efd2a\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") "
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.698634    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-dshm\") pod \"714c801b-8384-450a-bef2-53760f5efd2a\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") "
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.698699    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-tmp-dir\") pod \"714c801b-8384-450a-bef2-53760f5efd2a\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") "
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.698755    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-home\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-home\") pod \"714c801b-8384-450a-bef2-53760f5efd2a\" (UID: \"714c801b-8384-450a-bef2-53760f5efd2a\") "
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.700422    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-model-cache" pod "714c801b-8384-450a-bef2-53760f5efd2a" (UID: "714c801b-8384-450a-bef2-53760f5efd2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.700678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-tmp-dir" pod "714c801b-8384-450a-bef2-53760f5efd2a" (UID: "714c801b-8384-450a-bef2-53760f5efd2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.700729    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-home" pod "714c801b-8384-450a-bef2-53760f5efd2a" (UID: "714c801b-8384-450a-bef2-53760f5efd2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.701030    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.701049    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.702838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.701065    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.723788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.723679    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714c801b-8384-450a-bef2-53760f5efd2a-kube-api-access-qklfh" pod "714c801b-8384-450a-bef2-53760f5efd2a" (UID: "714c801b-8384-450a-bef2-53760f5efd2a"). InnerVolumeSpecName "kube-api-access-qklfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:00.729276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.729228    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714c801b-8384-450a-bef2-53760f5efd2a-tls-certs" pod "714c801b-8384-450a-bef2-53760f5efd2a" (UID: "714c801b-8384-450a-bef2-53760f5efd2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:00.731513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.731458    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-dshm" pod "714c801b-8384-450a-bef2-53760f5efd2a" (UID: "714c801b-8384-450a-bef2-53760f5efd2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.765536 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.761223    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 20:30:00.802316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.801612    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25a5898-dfb6-4257-935d-e4290aa91888-tls-certs\" (UniqueName: \"kubernetes.io/secret/a25a5898-dfb6-4257-935d-e4290aa91888-tls-certs\") pod \"a25a5898-dfb6-4257-935d-e4290aa91888\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") "
Oct 30 20:30:00.802316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.801752    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-home\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-home\") pod \"a25a5898-dfb6-4257-935d-e4290aa91888\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") "
Oct 30 20:30:00.802316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.801810    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-tmp-dir\") pod \"a25a5898-dfb6-4257-935d-e4290aa91888\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") "
Oct 30 20:30:00.802316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.801867    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-dshm\") pod \"a25a5898-dfb6-4257-935d-e4290aa91888\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") "
Oct 30 20:30:00.802316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.801942    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25a5898-dfb6-4257-935d-e4290aa91888-kube-api-access-g4b2k\" (UniqueName: \"kubernetes.io/projected/a25a5898-dfb6-4257-935d-e4290aa91888-kube-api-access-g4b2k\") pod \"a25a5898-dfb6-4257-935d-e4290aa91888\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") "
Oct 30 20:30:00.803215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.803035    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-tmp-dir" pod "a25a5898-dfb6-4257-935d-e4290aa91888" (UID: "a25a5898-dfb6-4257-935d-e4290aa91888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.804303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.803362    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-home" pod "a25a5898-dfb6-4257-935d-e4290aa91888" (UID: "a25a5898-dfb6-4257-935d-e4290aa91888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.804303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.804006    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-model-cache\") pod \"a25a5898-dfb6-4257-935d-e4290aa91888\" (UID: \"a25a5898-dfb6-4257-935d-e4290aa91888\") "
Oct 30 20:30:00.804804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.804719    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c801b-8384-450a-bef2-53760f5efd2a-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.804804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.804777    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.804981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.804792    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.804981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.804838    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklfh\" (UniqueName: \"kubernetes.io/projected/714c801b-8384-450a-bef2-53760f5efd2a-kube-api-access-qklfh\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.804981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.804852    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714c801b-8384-450a-bef2-53760f5efd2a-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.804981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.804720    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-model-cache" pod "a25a5898-dfb6-4257-935d-e4290aa91888" (UID: "a25a5898-dfb6-4257-935d-e4290aa91888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.826113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.823338    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25a5898-dfb6-4257-935d-e4290aa91888-kube-api-access-g4b2k" pod "a25a5898-dfb6-4257-935d-e4290aa91888" (UID: "a25a5898-dfb6-4257-935d-e4290aa91888"). InnerVolumeSpecName "kube-api-access-g4b2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:00.827457 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.826614    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-dshm" pod "a25a5898-dfb6-4257-935d-e4290aa91888" (UID: "a25a5898-dfb6-4257-935d-e4290aa91888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:00.830451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.827708    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25a5898-dfb6-4257-935d-e4290aa91888-tls-certs" pod "a25a5898-dfb6-4257-935d-e4290aa91888" (UID: "a25a5898-dfb6-4257-935d-e4290aa91888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:00.897192 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.897158    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:30:00.905939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.905905    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25a5898-dfb6-4257-935d-e4290aa91888-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.905939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.905938    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.906159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.905948    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4b2k\" (UniqueName: \"kubernetes.io/projected/a25a5898-dfb6-4257-935d-e4290aa91888-kube-api-access-g4b2k\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.906159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.905958    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25a5898-dfb6-4257-935d-e4290aa91888-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:00.974650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:00.974618    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:30:01.006503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006461    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-openshift-ca\") pod \"02e58824-5528-42b1-9d5c-3852f6876cd5\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") "
Oct 30 20:30:01.006690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006520    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-openshift-ca\") pod \"a5024961-e814-4fb2-953a-38f1920d7a08\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") "
Oct 30 20:30:01.006690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006575    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e58824-5528-42b1-9d5c-3852f6876cd5-kube-api-access-6fmhk\" (UniqueName: \"kubernetes.io/projected/02e58824-5528-42b1-9d5c-3852f6876cd5-kube-api-access-6fmhk\") pod \"02e58824-5528-42b1-9d5c-3852f6876cd5\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") "
Oct 30 20:30:01.006821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006738    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5024961-e814-4fb2-953a-38f1920d7a08-kube-api-access-z78jz\" (UniqueName: \"kubernetes.io/projected/a5024961-e814-4fb2-953a-38f1920d7a08-kube-api-access-z78jz\") pod \"a5024961-e814-4fb2-953a-38f1920d7a08\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") "
Oct 30 20:30:01.006821 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006776    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-xks-ca\") pod \"02e58824-5528-42b1-9d5c-3852f6876cd5\" (UID: \"02e58824-5528-42b1-9d5c-3852f6876cd5\") "
Oct 30 20:30:01.006933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006829    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-xks-ca\") pod \"a5024961-e814-4fb2-953a-38f1920d7a08\" (UID: \"a5024961-e814-4fb2-953a-38f1920d7a08\") "
Oct 30 20:30:01.006933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006859    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-openshift-ca" pod "a5024961-e814-4fb2-953a-38f1920d7a08" (UID: "a5024961-e814-4fb2-953a-38f1920d7a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:01.006933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.006862    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-openshift-ca" pod "02e58824-5528-42b1-9d5c-3852f6876cd5" (UID: "02e58824-5528-42b1-9d5c-3852f6876cd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:01.007370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.007343    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:01.007370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.007373    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:01.010011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.009979    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-xks-ca" pod "a5024961-e814-4fb2-953a-38f1920d7a08" (UID: "a5024961-e814-4fb2-953a-38f1920d7a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:01.010970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.010943    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-xks-ca" pod "02e58824-5528-42b1-9d5c-3852f6876cd5" (UID: "02e58824-5528-42b1-9d5c-3852f6876cd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:01.011092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.010974    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5024961-e814-4fb2-953a-38f1920d7a08-kube-api-access-z78jz" pod "a5024961-e814-4fb2-953a-38f1920d7a08" (UID: "a5024961-e814-4fb2-953a-38f1920d7a08"). InnerVolumeSpecName "kube-api-access-z78jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:01.011092 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.011005    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e58824-5528-42b1-9d5c-3852f6876cd5-kube-api-access-6fmhk" pod "02e58824-5528-42b1-9d5c-3852f6876cd5" (UID: "02e58824-5528-42b1-9d5c-3852f6876cd5"). InnerVolumeSpecName "kube-api-access-6fmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:01.108950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.108908    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fmhk\" (UniqueName: \"kubernetes.io/projected/02e58824-5528-42b1-9d5c-3852f6876cd5-kube-api-access-6fmhk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:01.108950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.108952    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z78jz\" (UniqueName: \"kubernetes.io/projected/a5024961-e814-4fb2-953a-38f1920d7a08-kube-api-access-z78jz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:01.109147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.108968    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e58824-5528-42b1-9d5c-3852f6876cd5-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:01.109147 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.108983    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5024961-e814-4fb2-953a-38f1920d7a08-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:01.146908 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:01.146869    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a0eed0_9ab5_4968_8ba2_09b4b521ab90.slice/crio-b40f14d5e41a5ff35bccf4663bac2a481a33cc5c1a550608c25fc2b9be8b7e9f WatchSource:0}: Error finding container b40f14d5e41a5ff35bccf4663bac2a481a33cc5c1a550608c25fc2b9be8b7e9f: Status 404 returned error can't find the container with id b40f14d5e41a5ff35bccf4663bac2a481a33cc5c1a550608c25fc2b9be8b7e9f
Oct 30 20:30:01.148064 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.148032    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"]
Oct 30 20:30:01.232549 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:01.232519    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbcab0c_a105_4dd1_a417_f324ba5bf305.slice/crio-9d4d303837465953412ff18e7b831904608cc0f6af4ac8486c97b28ae6c2734b WatchSource:0}: Error finding container 9d4d303837465953412ff18e7b831904608cc0f6af4ac8486c97b28ae6c2734b: Status 404 returned error can't find the container with id 9d4d303837465953412ff18e7b831904608cc0f6af4ac8486c97b28ae6c2734b
Oct 30 20:30:01.234644 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.234613    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"]
Oct 30 20:30:01.250717 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:01.250686    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd038ac_baa9_4b45_ad00_aef5cdb8127e.slice/crio-642c9c596a37fdde45965b081f6d2bb2dd11dcd3269f164c1905d31cb6fd1bcb WatchSource:0}: Error finding container 642c9c596a37fdde45965b081f6d2bb2dd11dcd3269f164c1905d31cb6fd1bcb: Status 404 returned error can't find the container with id 642c9c596a37fdde45965b081f6d2bb2dd11dcd3269f164c1905d31cb6fd1bcb
Oct 30 20:30:01.253088 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.253063    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"]
Oct 30 20:30:01.287358 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.287324    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"]
Oct 30 20:30:01.289232 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:01.289205    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c434ed_d6fa_4857_9642_f0627666212e.slice/crio-02cf507ebe0c5a24fe5b38d35a6109b4a7fab20a36d1c431cdfa4b52d2d9d277 WatchSource:0}: Error finding container 02cf507ebe0c5a24fe5b38d35a6109b4a7fab20a36d1c431cdfa4b52d2d9d277: Status 404 returned error can't find the container with id 02cf507ebe0c5a24fe5b38d35a6109b4a7fab20a36d1c431cdfa4b52d2d9d277
Oct 30 20:30:01.289476 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:01.289437    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ba0576_c136_4bff_ba0d_925da9b0488e.slice/crio-8b3b50f8cf003a6b86f67e193ca30e1dafa5b2b24586bac1b0d84e2438ed655a WatchSource:0}: Error finding container 8b3b50f8cf003a6b86f67e193ca30e1dafa5b2b24586bac1b0d84e2438ed655a: Status 404 returned error can't find the container with id 8b3b50f8cf003a6b86f67e193ca30e1dafa5b2b24586bac1b0d84e2438ed655a
Oct 30 20:30:01.293057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.293037    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"]
Oct 30 20:30:01.296732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.296709    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"]
Oct 30 20:30:01.306918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.306891    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"]
Oct 30 20:30:01.315159 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:01.315133    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42663d1a_4550_41bb_920b_52fdeb79efee.slice/crio-c36bbfe7599fbfa51ebd15fd5749d47377e33cf2faead99a7672e7fca3b45aca WatchSource:0}: Error finding container c36bbfe7599fbfa51ebd15fd5749d47377e33cf2faead99a7672e7fca3b45aca: Status 404 returned error can't find the container with id c36bbfe7599fbfa51ebd15fd5749d47377e33cf2faead99a7672e7fca3b45aca
Oct 30 20:30:01.545970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.545927    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" event={"ID":"13ba0576-c136-4bff-ba0d-925da9b0488e","Type":"ContainerStarted","Data":"8b3b50f8cf003a6b86f67e193ca30e1dafa5b2b24586bac1b0d84e2438ed655a"}
Oct 30 20:30:01.547787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.547762    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"
Oct 30 20:30:01.552096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.549876    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m" event={"ID":"02e58824-5528-42b1-9d5c-3852f6876cd5","Type":"ContainerDied","Data":"a79fd0049352995f8a60fa8881f907894aaccf46d86c13487734a4b410164dc9"}
Oct 30 20:30:01.552096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.549939    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a79fd0049352995f8a60fa8881f907894aaccf46d86c13487734a4b410164dc9"
Oct 30 20:30:01.552096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.550162    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" event={"ID":"53a0eed0-9ab5-4968-8ba2-09b4b521ab90","Type":"ContainerStarted","Data":"b40f14d5e41a5ff35bccf4663bac2a481a33cc5c1a550608c25fc2b9be8b7e9f"}
Oct 30 20:30:01.552096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.551850    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerStarted","Data":"02cf507ebe0c5a24fe5b38d35a6109b4a7fab20a36d1c431cdfa4b52d2d9d277"}
Oct 30 20:30:01.553241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.553211    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" event={"ID":"5ffd813f-fc60-482f-8268-f32c32262e8d","Type":"ContainerStarted","Data":"2c810150c6c0cfeaee7f74a4788ec1332000bdde5dfdd74cd42f715005fcdaf2"}
Oct 30 20:30:01.554539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.554516    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" event={"ID":"2dd038ac-baa9-4b45-ad00-aef5cdb8127e","Type":"ContainerStarted","Data":"642c9c596a37fdde45965b081f6d2bb2dd11dcd3269f164c1905d31cb6fd1bcb"}
Oct 30 20:30:01.556085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.556056    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j" event={"ID":"a25a5898-dfb6-4257-935d-e4290aa91888","Type":"ContainerDied","Data":"9c2f0f2f0375e0935fc54212711632535a83ce79e7f1ff0242fe1fe1f82eb5b0"}
Oct 30 20:30:01.556190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.556093    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"
Oct 30 20:30:01.556190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.556108    4519 scope.go:122] "RemoveContainer" containerID="9590bfa4a5c5602002f0a77ff8a461d61ea9c5fac43c0e011ac945f2b838b77b"
Oct 30 20:30:01.557713 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.557688    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" event={"ID":"42663d1a-4550-41bb-920b-52fdeb79efee","Type":"ContainerStarted","Data":"c36bbfe7599fbfa51ebd15fd5749d47377e33cf2faead99a7672e7fca3b45aca"}
Oct 30 20:30:01.559326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.559303    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc" event={"ID":"714c801b-8384-450a-bef2-53760f5efd2a","Type":"ContainerDied","Data":"e24d33da6a2b5a3bcc8949382de1b0164534a9bf5e60a6f303e11aab843d6385"}
Oct 30 20:30:01.559571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.559313    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"
Oct 30 20:30:01.561210 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.561186    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp" event={"ID":"a5024961-e814-4fb2-953a-38f1920d7a08","Type":"ContainerDied","Data":"e25a63d39989adcf7ea8059add402c5e53d067e0e0f7a32f5c5d6b619fb9f62c"}
Oct 30 20:30:01.561327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.561233    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e25a63d39989adcf7ea8059add402c5e53d067e0e0f7a32f5c5d6b619fb9f62c"
Oct 30 20:30:01.561327 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.561312    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"
Oct 30 20:30:01.565501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.565443    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" event={"ID":"bcbcab0c-a105-4dd1-a417-f324ba5bf305","Type":"ContainerStarted","Data":"9d4d303837465953412ff18e7b831904608cc0f6af4ac8486c97b28ae6c2734b"}
Oct 30 20:30:01.573184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.573122    4519 scope.go:122] "RemoveContainer" containerID="4f92532538612876c8b604a2fcfb8a8c134ea860ea7b543e4f985ff5e679ca6f"
Oct 30 20:30:01.628893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.628868    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"]
Oct 30 20:30:01.635257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.635228    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/simulator-w2-284bf6-kserve-74d9d5d5c7-7kg2j"]
Oct 30 20:30:01.684359 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.684326    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"]
Oct 30 20:30:01.688192 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.688164    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/premium-w2-284bf6-kserve-5795d7cb-5zpqc"]
Oct 30 20:30:01.887570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.887536    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"]
Oct 30 20:30:01.888292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888259    4519 cpu_manager.go:414] "containerMap: removing container" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main"
Oct 30 20:30:01.888292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888287    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main"
Oct 30 20:30:01.888581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888313    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a5024961-e814-4fb2-953a-38f1920d7a08" containerName="revoke-keys"
Oct 30 20:30:01.888581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888322    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5024961-e814-4fb2-953a-38f1920d7a08" containerName="revoke-keys"
Oct 30 20:30:01.888581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888374    4519 cpu_manager.go:414] "containerMap: removing container" podUID="02e58824-5528-42b1-9d5c-3852f6876cd5" containerName="revoke-keys"
Oct 30 20:30:01.888581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888384    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e58824-5528-42b1-9d5c-3852f6876cd5" containerName="revoke-keys"
Oct 30 20:30:01.888581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888416    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main"
Oct 30 20:30:01.888581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888424    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main"
Oct 30 20:30:01.888931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888594    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5024961-e814-4fb2-953a-38f1920d7a08" containerName="revoke-keys"
Oct 30 20:30:01.888931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888621    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e58824-5528-42b1-9d5c-3852f6876cd5" containerName="revoke-keys"
Oct 30 20:30:01.888931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888636    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" containerName="main"
Oct 30 20:30:01.888931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.888645    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="714c801b-8384-450a-bef2-53760f5efd2a" containerName="main"
Oct 30 20:30:01.892223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.892197    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.908664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.908615    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-77ac5a-openshift-default-dockercfg-497sf\""
Oct 30 20:30:01.917524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.917459    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-kube-api-access-k2kkp\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.917652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.917587    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-envoy\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.917652 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.917646    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-certs\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.917762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.917669    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360416d1-6d52-4c39-9f6a-158df599e5a1-istiod-ca-cert\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.917834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.917809    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-data\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.917872 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.917855    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-token\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.918006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.917983    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.918112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.918045    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-credential-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.918187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.918167    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-podinfo\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:01.927505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:01.927437    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"]
Oct 30 20:30:02.019560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019454    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-kube-api-access-k2kkp\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019536    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-envoy\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019585    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-certs\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019607    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360416d1-6d52-4c39-9f6a-158df599e5a1-istiod-ca-cert\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019643    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-data\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019665    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-token\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019701    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019733    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-credential-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.019803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.019795    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-podinfo\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.020169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.020037    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-certs\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.020285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.020262    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-credential-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.020364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.020334    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.020503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.020458    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360416d1-6d52-4c39-9f6a-158df599e5a1-istiod-ca-cert\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.020663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.020513    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-data\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.022116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.022089    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-envoy\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.022369 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.022351    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-podinfo\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.040172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.040141    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-token\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.040320 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.040299    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-kube-api-access-k2kkp\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.208196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.208159    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:02.227043 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:02.226960    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:30:02.553032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.552993    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"]
Oct 30 20:30:02.578922 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.578889    4519 generic.go:356] "Generic (PLEG): container finished" podID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerID="9512dec518f7580a82470d80b74e34457dc96ae9a26948413b522f168b198a6b" exitCode=60
Oct 30 20:30:02.579081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.578972    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" event={"ID":"5ffd813f-fc60-482f-8268-f32c32262e8d","Type":"ContainerDied","Data":"9512dec518f7580a82470d80b74e34457dc96ae9a26948413b522f168b198a6b"}
Oct 30 20:30:02.579269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.579254    4519 scope.go:122] "RemoveContainer" containerID="9512dec518f7580a82470d80b74e34457dc96ae9a26948413b522f168b198a6b"
Oct 30 20:30:02.580565 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:02.580536    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360416d1_6d52_4c39_9f6a_158df599e5a1.slice/crio-4476ab934734364852bcc317a93379ce083be6c4b126f41f95a6a6f569b62a32 WatchSource:0}: Error finding container 4476ab934734364852bcc317a93379ce083be6c4b126f41f95a6a6f569b62a32: Status 404 returned error can't find the container with id 4476ab934734364852bcc317a93379ce083be6c4b126f41f95a6a6f569b62a32
Oct 30 20:30:02.581607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.581573    4519 generic.go:356] "Generic (PLEG): container finished" podID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerID="dbd9db685823f19fd2dce7f29eac798e8e8a3e6ac116acee6f2116181c61888d" exitCode=60
Oct 30 20:30:02.581671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.581612    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" event={"ID":"2dd038ac-baa9-4b45-ad00-aef5cdb8127e","Type":"ContainerDied","Data":"dbd9db685823f19fd2dce7f29eac798e8e8a3e6ac116acee6f2116181c61888d"}
Oct 30 20:30:02.581953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.581931    4519 scope.go:122] "RemoveContainer" containerID="dbd9db685823f19fd2dce7f29eac798e8e8a3e6ac116acee6f2116181c61888d"
Oct 30 20:30:02.583684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.583657    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:30:02.583769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.583724    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:30:02.583846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.583768    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:30:02.584874 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.584851    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" event={"ID":"42663d1a-4550-41bb-920b-52fdeb79efee","Type":"ContainerStarted","Data":"1c162779b889b5f62057f9be132947b51d96d9b11336275e89d0339d4a7cec2a"}
Oct 30 20:30:02.588151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.588127    4519 generic.go:356] "Generic (PLEG): container finished" podID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerID="8cf22ca7f3a2bc9d54d471c8b783b997eca0ba6888489895cbd3c25566c8d193" exitCode=60
Oct 30 20:30:02.588270 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.588210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" event={"ID":"bcbcab0c-a105-4dd1-a417-f324ba5bf305","Type":"ContainerDied","Data":"8cf22ca7f3a2bc9d54d471c8b783b997eca0ba6888489895cbd3c25566c8d193"}
Oct 30 20:30:02.588371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.588355    4519 scope.go:122] "RemoveContainer" containerID="8cf22ca7f3a2bc9d54d471c8b783b997eca0ba6888489895cbd3c25566c8d193"
Oct 30 20:30:02.592205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.591478    4519 generic.go:356] "Generic (PLEG): container finished" podID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerID="5cdb6af580a8cd41157d9eaa6e6c94e1bfacf874fadb636abc7d74413388b31c" exitCode=60
Oct 30 20:30:02.592205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.591627    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" event={"ID":"13ba0576-c136-4bff-ba0d-925da9b0488e","Type":"ContainerDied","Data":"5cdb6af580a8cd41157d9eaa6e6c94e1bfacf874fadb636abc7d74413388b31c"}
Oct 30 20:30:02.592205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.591931    4519 scope.go:122] "RemoveContainer" containerID="5cdb6af580a8cd41157d9eaa6e6c94e1bfacf874fadb636abc7d74413388b31c"
Oct 30 20:30:02.595560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.595538    4519 generic.go:356] "Generic (PLEG): container finished" podID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerID="1738c44eef055e5651aa700d356c229e85c1bd1769f06286b937cb0e4330acb9" exitCode=60
Oct 30 20:30:02.595666 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.595618    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" event={"ID":"53a0eed0-9ab5-4968-8ba2-09b4b521ab90","Type":"ContainerDied","Data":"1738c44eef055e5651aa700d356c229e85c1bd1769f06286b937cb0e4330acb9"}
Oct 30 20:30:02.595788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.595767    4519 scope.go:122] "RemoveContainer" containerID="1738c44eef055e5651aa700d356c229e85c1bd1769f06286b937cb0e4330acb9"
Oct 30 20:30:02.597477 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.597437    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerStarted","Data":"8eaa3cda912b33a39b1b8b7d51afea9749feaea5b63bdf6e4f10741968449398"}
Oct 30 20:30:02.771302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.771250    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" podStartSLOduration=2.771235596 podStartE2EDuration="2.771235596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:02.770137035 +0000 UTC m=+3327.270840758" watchObservedRunningTime="2026-10-30 20:30:02.771235596 +0000 UTC m=+3327.271939317"
Oct 30 20:30:02.920803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:02.920733    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" podStartSLOduration=2.920711948 podStartE2EDuration="2.920711948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:02.914119957 +0000 UTC m=+3327.414823681" watchObservedRunningTime="2026-10-30 20:30:02.920711948 +0000 UTC m=+3327.421415674"
Oct 30 20:30:03.067419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.067354    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"]
Oct 30 20:30:03.073983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.073733    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b336b3b-wt2tt"]
Oct 30 20:30:03.611074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.611035    4519 generic.go:356] "Generic (PLEG): container finished" podID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerID="5223f4139d4d689160b3b581c7ae36be118e846924d6480172d1cb412476a300" exitCode=60
Oct 30 20:30:03.611538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.611335    4519 scope.go:122] "RemoveContainer" containerID="5223f4139d4d689160b3b581c7ae36be118e846924d6480172d1cb412476a300"
Oct 30 20:30:03.611968 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:03.611574    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg_odh-ai-gateway-infra(bcbcab0c-a105-4dd1-a417-f324ba5bf305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305"
Oct 30 20:30:03.613419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.613386    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075d9333-ef02-4c2d-8d9b-03551f02a8e7" path="/var/lib/kubelet/pods/075d9333-ef02-4c2d-8d9b-03551f02a8e7/volumes"
Oct 30 20:30:03.613955 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.613933    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714c801b-8384-450a-bef2-53760f5efd2a" path="/var/lib/kubelet/pods/714c801b-8384-450a-bef2-53760f5efd2a/volumes"
Oct 30 20:30:03.614540 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.614518    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25a5898-dfb6-4257-935d-e4290aa91888" path="/var/lib/kubelet/pods/a25a5898-dfb6-4257-935d-e4290aa91888/volumes"
Oct 30 20:30:03.614900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.614881    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" event={"ID":"360416d1-6d52-4c39-9f6a-158df599e5a1","Type":"ContainerStarted","Data":"9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666"}
Oct 30 20:30:03.614973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.614922    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" event={"ID":"360416d1-6d52-4c39-9f6a-158df599e5a1","Type":"ContainerStarted","Data":"4476ab934734364852bcc317a93379ce083be6c4b126f41f95a6a6f569b62a32"}
Oct 30 20:30:03.614973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.614953    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" event={"ID":"bcbcab0c-a105-4dd1-a417-f324ba5bf305","Type":"ContainerDied","Data":"5223f4139d4d689160b3b581c7ae36be118e846924d6480172d1cb412476a300"}
Oct 30 20:30:03.615086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.614987    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" event={"ID":"13ba0576-c136-4bff-ba0d-925da9b0488e","Type":"ContainerStarted","Data":"e75a988ab6af2037fbe54854abce6c02d2d060586dce547ea9342e6c86efb7ad"}
Oct 30 20:30:03.615086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.615021    4519 scope.go:122] "RemoveContainer" containerID="8cf22ca7f3a2bc9d54d471c8b783b997eca0ba6888489895cbd3c25566c8d193"
Oct 30 20:30:03.615716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.615696    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" event={"ID":"53a0eed0-9ab5-4968-8ba2-09b4b521ab90","Type":"ContainerStarted","Data":"e8d10558f7030b49e06564d2bc7c396c2a5a1e897d3c70aaaf11223990d068fc"}
Oct 30 20:30:03.617434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.617409    4519 generic.go:356] "Generic (PLEG): container finished" podID="02c434ed-d6fa-4857-9642-f0627666212e" containerID="8eaa3cda912b33a39b1b8b7d51afea9749feaea5b63bdf6e4f10741968449398" exitCode=60
Oct 30 20:30:03.617558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.617493    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerDied","Data":"8eaa3cda912b33a39b1b8b7d51afea9749feaea5b63bdf6e4f10741968449398"}
Oct 30 20:30:03.617693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.617675    4519 scope.go:122] "RemoveContainer" containerID="8eaa3cda912b33a39b1b8b7d51afea9749feaea5b63bdf6e4f10741968449398"
Oct 30 20:30:03.619919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.619551    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" event={"ID":"5ffd813f-fc60-482f-8268-f32c32262e8d","Type":"ContainerStarted","Data":"f4481bd7e5a6059f2b8e92c5b2d2544341353cbb46280215ac1f7b7da7e47a4f"}
Oct 30 20:30:03.621712 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.621689    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" event={"ID":"2dd038ac-baa9-4b45-ad00-aef5cdb8127e","Type":"ContainerStarted","Data":"f2f9438291e8e399f9c5738049e94dca42f170654bfbe21d3e23a79fb019a186"}
Oct 30 20:30:03.623437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.623410    4519 generic.go:356] "Generic (PLEG): container finished" podID="42663d1a-4550-41bb-920b-52fdeb79efee" containerID="1c162779b889b5f62057f9be132947b51d96d9b11336275e89d0339d4a7cec2a" exitCode=60
Oct 30 20:30:03.623546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.623446    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" event={"ID":"42663d1a-4550-41bb-920b-52fdeb79efee","Type":"ContainerDied","Data":"1c162779b889b5f62057f9be132947b51d96d9b11336275e89d0339d4a7cec2a"}
Oct 30 20:30:03.623693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.623676    4519 scope.go:122] "RemoveContainer" containerID="1c162779b889b5f62057f9be132947b51d96d9b11336275e89d0339d4a7cec2a"
Oct 30 20:30:03.689067 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.688998    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" podStartSLOduration=2.6889787099999998 podStartE2EDuration="2.68897871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.686562565 +0000 UTC m=+3328.187266297" watchObservedRunningTime="2026-10-30 20:30:03.68897871 +0000 UTC m=+3328.189682435"
Oct 30 20:30:03.825300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.825233    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" podStartSLOduration=3.825217188 podStartE2EDuration="3.825217188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.822347158 +0000 UTC m=+3328.323050883" watchObservedRunningTime="2026-10-30 20:30:03.825217188 +0000 UTC m=+3328.325920912"
Oct 30 20:30:03.915496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:03.911122    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" podStartSLOduration=3.911102623 podStartE2EDuration="3.911102623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.909612895 +0000 UTC m=+3328.410316620" watchObservedRunningTime="2026-10-30 20:30:03.911102623 +0000 UTC m=+3328.411806349"
Oct 30 20:30:04.115227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.115197    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"]
Oct 30 20:30:04.123763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.123738    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7f94-n55nz"]
Oct 30 20:30:04.209987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.209286    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:04.211642 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.211598    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Oct 30 20:30:04.211803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.211681    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Oct 30 20:30:04.283491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.282972    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" podStartSLOduration=4.282950505 podStartE2EDuration="4.282950505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:04.181057904 +0000 UTC m=+3328.681761629" watchObservedRunningTime="2026-10-30 20:30:04.282950505 +0000 UTC m=+3328.783654230"
Oct 30 20:30:04.288133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.284480    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" podStartSLOduration=4.284453418 podStartE2EDuration="4.284453418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:04.245553423 +0000 UTC m=+3328.746257147" watchObservedRunningTime="2026-10-30 20:30:04.284453418 +0000 UTC m=+3328.785157142"
Oct 30 20:30:04.634208 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.634165    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" event={"ID":"42663d1a-4550-41bb-920b-52fdeb79efee","Type":"ContainerStarted","Data":"93ead50f68ed083fd7c0943d8dacc514f5fa887ffb78e7e227e62c4eda5599f8"}
Oct 30 20:30:04.636908 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.636877    4519 scope.go:122] "RemoveContainer" containerID="5223f4139d4d689160b3b581c7ae36be118e846924d6480172d1cb412476a300"
Oct 30 20:30:04.637213 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:04.637186    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg_odh-ai-gateway-infra(bcbcab0c-a105-4dd1-a417-f324ba5bf305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305"
Oct 30 20:30:04.638803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.638772    4519 generic.go:356] "Generic (PLEG): container finished" podID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerID="e75a988ab6af2037fbe54854abce6c02d2d060586dce547ea9342e6c86efb7ad" exitCode=60
Oct 30 20:30:04.638911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.638851    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" event={"ID":"13ba0576-c136-4bff-ba0d-925da9b0488e","Type":"ContainerDied","Data":"e75a988ab6af2037fbe54854abce6c02d2d060586dce547ea9342e6c86efb7ad"}
Oct 30 20:30:04.638969 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.638918    4519 scope.go:122] "RemoveContainer" containerID="5cdb6af580a8cd41157d9eaa6e6c94e1bfacf874fadb636abc7d74413388b31c"
Oct 30 20:30:04.639112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.639091    4519 scope.go:122] "RemoveContainer" containerID="e75a988ab6af2037fbe54854abce6c02d2d060586dce547ea9342e6c86efb7ad"
Oct 30 20:30:04.639448 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:04.639373    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx_odh-ai-gateway-infra(13ba0576-c136-4bff-ba0d-925da9b0488e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e"
Oct 30 20:30:04.641165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.641142    4519 generic.go:356] "Generic (PLEG): container finished" podID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerID="e8d10558f7030b49e06564d2bc7c396c2a5a1e897d3c70aaaf11223990d068fc" exitCode=60
Oct 30 20:30:04.641283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.641211    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" event={"ID":"53a0eed0-9ab5-4968-8ba2-09b4b521ab90","Type":"ContainerDied","Data":"e8d10558f7030b49e06564d2bc7c396c2a5a1e897d3c70aaaf11223990d068fc"}
Oct 30 20:30:04.641449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.641432    4519 scope.go:122] "RemoveContainer" containerID="e8d10558f7030b49e06564d2bc7c396c2a5a1e897d3c70aaaf11223990d068fc"
Oct 30 20:30:04.641740 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:04.641717    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx_odh-ai-gateway-infra(53a0eed0-9ab5-4968-8ba2-09b4b521ab90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90"
Oct 30 20:30:04.643282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.643259    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerStarted","Data":"7e7a601d8c040a9e60269cb960d851dcf1a6c682a72a49a4327401299c712832"}
Oct 30 20:30:04.645433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.645411    4519 generic.go:356] "Generic (PLEG): container finished" podID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerID="f4481bd7e5a6059f2b8e92c5b2d2544341353cbb46280215ac1f7b7da7e47a4f" exitCode=60
Oct 30 20:30:04.645580 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.645495    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" event={"ID":"5ffd813f-fc60-482f-8268-f32c32262e8d","Type":"ContainerDied","Data":"f4481bd7e5a6059f2b8e92c5b2d2544341353cbb46280215ac1f7b7da7e47a4f"}
Oct 30 20:30:04.645724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.645702    4519 scope.go:122] "RemoveContainer" containerID="f4481bd7e5a6059f2b8e92c5b2d2544341353cbb46280215ac1f7b7da7e47a4f"
Oct 30 20:30:04.645986 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:04.645968    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd_odh-ai-gateway-infra(5ffd813f-fc60-482f-8268-f32c32262e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d"
Oct 30 20:30:04.681725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.681674    4519 generic.go:356] "Generic (PLEG): container finished" podID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerID="f2f9438291e8e399f9c5738049e94dca42f170654bfbe21d3e23a79fb019a186" exitCode=60
Oct 30 20:30:04.681925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.681806    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" event={"ID":"2dd038ac-baa9-4b45-ad00-aef5cdb8127e","Type":"ContainerDied","Data":"f2f9438291e8e399f9c5738049e94dca42f170654bfbe21d3e23a79fb019a186"}
Oct 30 20:30:04.682397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.682379    4519 scope.go:122] "RemoveContainer" containerID="f2f9438291e8e399f9c5738049e94dca42f170654bfbe21d3e23a79fb019a186"
Oct 30 20:30:04.682696 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:04.682673    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x_odh-ai-gateway-infra(2dd038ac-baa9-4b45-ad00-aef5cdb8127e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e"
Oct 30 20:30:04.683022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.682997    4519 scope.go:122] "RemoveContainer" containerID="1738c44eef055e5651aa700d356c229e85c1bd1769f06286b937cb0e4330acb9"
Oct 30 20:30:04.699490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.699455    4519 scope.go:122] "RemoveContainer" containerID="9512dec518f7580a82470d80b74e34457dc96ae9a26948413b522f168b198a6b"
Oct 30 20:30:04.713958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:04.713935    4519 scope.go:122] "RemoveContainer" containerID="dbd9db685823f19fd2dce7f29eac798e8e8a3e6ac116acee6f2116181c61888d"
Oct 30 20:30:05.209353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.209313    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Oct 30 20:30:05.209618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.209374    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Oct 30 20:30:05.613241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.613151    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d6fdd3-558e-4629-a380-95ff34ff75ff" path="/var/lib/kubelet/pods/13d6fdd3-558e-4629-a380-95ff34ff75ff/volumes"
Oct 30 20:30:05.693737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.693701    4519 generic.go:356] "Generic (PLEG): container finished" podID="02c434ed-d6fa-4857-9642-f0627666212e" containerID="7e7a601d8c040a9e60269cb960d851dcf1a6c682a72a49a4327401299c712832" exitCode=60
Oct 30 20:30:05.694242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.693788    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerDied","Data":"7e7a601d8c040a9e60269cb960d851dcf1a6c682a72a49a4327401299c712832"}
Oct 30 20:30:05.694242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.693860    4519 scope.go:122] "RemoveContainer" containerID="8eaa3cda912b33a39b1b8b7d51afea9749feaea5b63bdf6e4f10741968449398"
Oct 30 20:30:05.694242 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.694143    4519 scope.go:122] "RemoveContainer" containerID="7e7a601d8c040a9e60269cb960d851dcf1a6c682a72a49a4327401299c712832"
Oct 30 20:30:05.694483 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:05.694449    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-b5rg2_odh-ai-gateway-infra(02c434ed-d6fa-4857-9642-f0627666212e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" podUID="02c434ed-d6fa-4857-9642-f0627666212e"
Oct 30 20:30:05.696736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.696716    4519 scope.go:122] "RemoveContainer" containerID="f4481bd7e5a6059f2b8e92c5b2d2544341353cbb46280215ac1f7b7da7e47a4f"
Oct 30 20:30:05.697130 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:05.697102    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd_odh-ai-gateway-infra(5ffd813f-fc60-482f-8268-f32c32262e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d"
Oct 30 20:30:05.698661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.698640    4519 scope.go:122] "RemoveContainer" containerID="f2f9438291e8e399f9c5738049e94dca42f170654bfbe21d3e23a79fb019a186"
Oct 30 20:30:05.698954 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:05.698933    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x_odh-ai-gateway-infra(2dd038ac-baa9-4b45-ad00-aef5cdb8127e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e"
Oct 30 20:30:05.700611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.700584    4519 generic.go:356] "Generic (PLEG): container finished" podID="42663d1a-4550-41bb-920b-52fdeb79efee" containerID="93ead50f68ed083fd7c0943d8dacc514f5fa887ffb78e7e227e62c4eda5599f8" exitCode=60
Oct 30 20:30:05.700762 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.700738    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" event={"ID":"42663d1a-4550-41bb-920b-52fdeb79efee","Type":"ContainerDied","Data":"93ead50f68ed083fd7c0943d8dacc514f5fa887ffb78e7e227e62c4eda5599f8"}
Oct 30 20:30:05.701321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.701298    4519 scope.go:122] "RemoveContainer" containerID="93ead50f68ed083fd7c0943d8dacc514f5fa887ffb78e7e227e62c4eda5599f8"
Oct 30 20:30:05.701644 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:05.701616    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9_odh-ai-gateway-infra(42663d1a-4550-41bb-920b-52fdeb79efee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" podUID="42663d1a-4550-41bb-920b-52fdeb79efee"
Oct 30 20:30:05.702936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.702914    4519 scope.go:122] "RemoveContainer" containerID="e75a988ab6af2037fbe54854abce6c02d2d060586dce547ea9342e6c86efb7ad"
Oct 30 20:30:05.703184 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:05.703162    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx_odh-ai-gateway-infra(13ba0576-c136-4bff-ba0d-925da9b0488e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e"
Oct 30 20:30:05.704855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.704836    4519 scope.go:122] "RemoveContainer" containerID="e8d10558f7030b49e06564d2bc7c396c2a5a1e897d3c70aaaf11223990d068fc"
Oct 30 20:30:05.705016 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:05.704999    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx_odh-ai-gateway-infra(53a0eed0-9ab5-4968-8ba2-09b4b521ab90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90"
Oct 30 20:30:05.716748 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:05.716731    4519 scope.go:122] "RemoveContainer" containerID="1c162779b889b5f62057f9be132947b51d96d9b11336275e89d0339d4a7cec2a"
Oct 30 20:30:06.209064 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:06.209027    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Oct 30 20:30:06.209274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:06.209092    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Oct 30 20:30:06.715260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:06.715224    4519 scope.go:122] "RemoveContainer" containerID="7e7a601d8c040a9e60269cb960d851dcf1a6c682a72a49a4327401299c712832"
Oct 30 20:30:06.715760 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:06.715661    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-b5rg2_odh-ai-gateway-infra(02c434ed-d6fa-4857-9642-f0627666212e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" podUID="02c434ed-d6fa-4857-9642-f0627666212e"
Oct 30 20:30:06.716978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:06.716955    4519 scope.go:122] "RemoveContainer" containerID="93ead50f68ed083fd7c0943d8dacc514f5fa887ffb78e7e227e62c4eda5599f8"
Oct 30 20:30:06.717179 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:06.717156    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9_odh-ai-gateway-infra(42663d1a-4550-41bb-920b-52fdeb79efee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" podUID="42663d1a-4550-41bb-920b-52fdeb79efee"
Oct 30 20:30:07.208698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:07.208610    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Oct 30 20:30:07.208903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:07.208728    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Oct 30 20:30:08.208815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:08.208780    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Oct 30 20:30:08.209251 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:08.208851    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Oct 30 20:30:08.317040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:08.316998    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"]
Oct 30 20:30:08.895489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:08.895224    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"]
Oct 30 20:30:09.359786 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.357754    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:09.359786 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.359345    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:09.361558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.361523    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:09.886446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.886407    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"]
Oct 30 20:30:09.893169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.893137    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:09.912918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.912888    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"]
Oct 30 20:30:09.913507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.913479    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:09.917992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.917454    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"]
Oct 30 20:30:09.917992 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.917584    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:09.967425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.967379    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"]
Oct 30 20:30:09.973978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.973935    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkvs\" (UniqueName: \"kubernetes.io/projected/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-kube-api-access-jrkvs\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:09.974139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.974079    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:09.974181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.974134    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:09.974265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.974211    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:09.974315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.974299    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:09.974374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:09.974361    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzz6\" (UniqueName: \"kubernetes.io/projected/cded2d35-8c43-4b09-aa44-24b0bc2600d5-kube-api-access-rfzz6\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.075106 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.075067    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkvs\" (UniqueName: \"kubernetes.io/projected/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-kube-api-access-jrkvs\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:10.075334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.075142    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:10.075334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.075179    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:10.075334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.075246    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.075334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.075325    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.075605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.075375    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzz6\" (UniqueName: \"kubernetes.io/projected/cded2d35-8c43-4b09-aa44-24b0bc2600d5-kube-api-access-rfzz6\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.076365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.076337    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:10.076560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.076536    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.078061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.078033    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.078332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.078306    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:10.100283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.100252    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkvs\" (UniqueName: \"kubernetes.io/projected/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-kube-api-access-jrkvs\") pod \"maas-api-revoke-sub-d1301367-w77z6\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:10.100694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.100669    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzz6\" (UniqueName: \"kubernetes.io/projected/cded2d35-8c43-4b09-aa44-24b0bc2600d5-kube-api-access-rfzz6\") pod \"maas-api-revoke-sub-f9ec88d4-5w74p\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.208757 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.208722    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:10.236760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.236722    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:10.531491 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:10.531436    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcded2d35_8c43_4b09_aa44_24b0bc2600d5.slice/crio-73ab3355a6f446ea46ade381f2e70b6bbc349cf3349891ef78a1d2f0e3b5a00b WatchSource:0}: Error finding container 73ab3355a6f446ea46ade381f2e70b6bbc349cf3349891ef78a1d2f0e3b5a00b: Status 404 returned error can't find the container with id 73ab3355a6f446ea46ade381f2e70b6bbc349cf3349891ef78a1d2f0e3b5a00b
Oct 30 20:30:10.532373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.532344    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"]
Oct 30 20:30:10.570620 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.570532    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"]
Oct 30 20:30:10.573687 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:10.573659    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617c0e40_57e3_4c02_b4f3_5a7c2ac7c55d.slice/crio-4635ef2de8d8065416556bdf7a92940c7122d0f323b20e2ce9545737f3c93ee9 WatchSource:0}: Error finding container 4635ef2de8d8065416556bdf7a92940c7122d0f323b20e2ce9545737f3c93ee9: Status 404 returned error can't find the container with id 4635ef2de8d8065416556bdf7a92940c7122d0f323b20e2ce9545737f3c93ee9
Oct 30 20:30:10.765202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.765167    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6" event={"ID":"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d","Type":"ContainerStarted","Data":"4635ef2de8d8065416556bdf7a92940c7122d0f323b20e2ce9545737f3c93ee9"}
Oct 30 20:30:10.768171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:10.768144    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p" event={"ID":"cded2d35-8c43-4b09-aa44-24b0bc2600d5","Type":"ContainerStarted","Data":"73ab3355a6f446ea46ade381f2e70b6bbc349cf3349891ef78a1d2f0e3b5a00b"}
Oct 30 20:30:11.801298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:11.801239    4519 generic.go:356] "Generic (PLEG): container finished" podID="617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" containerID="d7f0f306aec4737c6b0e997142bddd35e6ab95e52de712c70e5a9c491af0dc53" exitCode=0
Oct 30 20:30:11.802201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:11.801459    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6" event={"ID":"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d","Type":"ContainerDied","Data":"d7f0f306aec4737c6b0e997142bddd35e6ab95e52de712c70e5a9c491af0dc53"}
Oct 30 20:30:11.805428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:11.805165    4519 generic.go:356] "Generic (PLEG): container finished" podID="cded2d35-8c43-4b09-aa44-24b0bc2600d5" containerID="84c9b5ee07334f81e9b0d057a1569df1110bfba590f5abd35e7aa34494242e5f" exitCode=0
Oct 30 20:30:11.805428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:11.805266    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p" event={"ID":"cded2d35-8c43-4b09-aa44-24b0bc2600d5","Type":"ContainerDied","Data":"84c9b5ee07334f81e9b0d057a1569df1110bfba590f5abd35e7aa34494242e5f"}
Oct 30 20:30:11.890543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:11.887371    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"]
Oct 30 20:30:12.006648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.005293    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"]
Oct 30 20:30:12.006648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.005568    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9" podUID="6f594f2c-af03-4c8d-98c5-4d3ae876650a" containerName="payload-pre-processing" containerID="cri-o://ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6" gracePeriod=30
Oct 30 20:30:12.180694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.180654    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:12.358488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.354665    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"]
Oct 30 20:30:12.359591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.359558    4519 cpu_manager.go:414] "containerMap: removing container" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerName="cleanup"
Oct 30 20:30:12.359791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.359776    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerName="cleanup"
Oct 30 20:30:12.368545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.359904    4519 cpu_manager.go:414] "containerMap: removing container" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerName="cleanup"
Oct 30 20:30:12.368773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.368743    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerName="cleanup"
Oct 30 20:30:12.369203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.369176    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerName="cleanup"
Oct 30 20:30:12.406715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.406677    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ba0576-c136-4bff-ba0d-925da9b0488e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ba0576-c136-4bff-ba0d-925da9b0488e-maas-api-service-ca\") pod \"13ba0576-c136-4bff-ba0d-925da9b0488e\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") "
Oct 30 20:30:12.408439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.408406    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ba0576-c136-4bff-ba0d-925da9b0488e-kube-api-access-mgxsl\" (UniqueName: \"kubernetes.io/projected/13ba0576-c136-4bff-ba0d-925da9b0488e-kube-api-access-mgxsl\") pod \"13ba0576-c136-4bff-ba0d-925da9b0488e\" (UID: \"13ba0576-c136-4bff-ba0d-925da9b0488e\") "
Oct 30 20:30:12.410441 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.410411    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ba0576-c136-4bff-ba0d-925da9b0488e-maas-api-service-ca" pod "13ba0576-c136-4bff-ba0d-925da9b0488e" (UID: "13ba0576-c136-4bff-ba0d-925da9b0488e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:12.429771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.429741    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ba0576-c136-4bff-ba0d-925da9b0488e-kube-api-access-mgxsl" pod "13ba0576-c136-4bff-ba0d-925da9b0488e" (UID: "13ba0576-c136-4bff-ba0d-925da9b0488e"). InnerVolumeSpecName "kube-api-access-mgxsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:12.438948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.438873    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.440201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.440166    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"]
Oct 30 20:30:12.498887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.498856    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:30:12.510758 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.510722    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f594f2c-af03-4c8d-98c5-4d3ae876650a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f594f2c-af03-4c8d-98c5-4d3ae876650a-plugins-config-volume\") pod \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") "
Oct 30 20:30:12.510948 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.510783    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f594f2c-af03-4c8d-98c5-4d3ae876650a-kube-api-access-92n8j\" (UniqueName: \"kubernetes.io/projected/6f594f2c-af03-4c8d-98c5-4d3ae876650a-kube-api-access-92n8j\") pod \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\" (UID: \"6f594f2c-af03-4c8d-98c5-4d3ae876650a\") "
Oct 30 20:30:12.511095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.511060    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f594f2c-af03-4c8d-98c5-4d3ae876650a-plugins-config-volume" pod "6f594f2c-af03-4c8d-98c5-4d3ae876650a" (UID: "6f594f2c-af03-4c8d-98c5-4d3ae876650a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:12.511212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.511146    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x2wd\" (UniqueName: \"kubernetes.io/projected/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-kube-api-access-7x2wd\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.511314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.511301    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.511454 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.511434    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.511607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.511591    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ba0576-c136-4bff-ba0d-925da9b0488e-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:12.511677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.511612    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f594f2c-af03-4c8d-98c5-4d3ae876650a-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:12.511677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.511628    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxsl\" (UniqueName: \"kubernetes.io/projected/13ba0576-c136-4bff-ba0d-925da9b0488e-kube-api-access-mgxsl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:12.512897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.512877    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f594f2c-af03-4c8d-98c5-4d3ae876650a-kube-api-access-92n8j" pod "6f594f2c-af03-4c8d-98c5-4d3ae876650a" (UID: "6f594f2c-af03-4c8d-98c5-4d3ae876650a"). InnerVolumeSpecName "kube-api-access-92n8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:12.613109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.613059    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.613300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.613207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.613300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.613263    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x2wd\" (UniqueName: \"kubernetes.io/projected/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-kube-api-access-7x2wd\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.613443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.613385    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92n8j\" (UniqueName: \"kubernetes.io/projected/6f594f2c-af03-4c8d-98c5-4d3ae876650a-kube-api-access-92n8j\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:12.614136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.614105    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.615918 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.615897    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.637625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.637597    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x2wd\" (UniqueName: \"kubernetes.io/projected/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-kube-api-access-7x2wd\") pod \"maas-api-revoke-sub-ad220342-7hrzf\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.791373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.791282    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"]
Oct 30 20:30:12.797169 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.797146    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:12.797632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.797592    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-843ea8-rjh6q"]
Oct 30 20:30:12.816103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.816075    4519 generic.go:356] "Generic (PLEG): container finished" podID="6f594f2c-af03-4c8d-98c5-4d3ae876650a" containerID="ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6" exitCode=0
Oct 30 20:30:12.816478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.816123    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"
Oct 30 20:30:12.816478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.816171    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9" event={"ID":"6f594f2c-af03-4c8d-98c5-4d3ae876650a","Type":"ContainerDied","Data":"ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6"}
Oct 30 20:30:12.816478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.816230    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9" event={"ID":"6f594f2c-af03-4c8d-98c5-4d3ae876650a","Type":"ContainerDied","Data":"4d7ed78696191d500a21a15024ac68f60d6c1b1f013c2ab62d2ac94330b7c588"}
Oct 30 20:30:12.816478 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.816265    4519 scope.go:122] "RemoveContainer" containerID="ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6"
Oct 30 20:30:12.818022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.818005    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"
Oct 30 20:30:12.818124 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.818090    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx" event={"ID":"13ba0576-c136-4bff-ba0d-925da9b0488e","Type":"ContainerDied","Data":"8b3b50f8cf003a6b86f67e193ca30e1dafa5b2b24586bac1b0d84e2438ed655a"}
Oct 30 20:30:12.833678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.833568    4519 scope.go:122] "RemoveContainer" containerID="ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6"
Oct 30 20:30:12.834135 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:12.833974    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6\": container with ID starting with ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6 not found: ID does not exist" containerID="ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6"
Oct 30 20:30:12.834135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.834014    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6"} err="failed to get container status \"ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6\": rpc error: code = NotFound desc = could not find container \"ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6\": container with ID starting with ef749e3382e8b95418f3a6e758641805eec822f621c94d3a8fefe2378943a5c6 not found: ID does not exist"
Oct 30 20:30:12.834135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.834038    4519 scope.go:122] "RemoveContainer" containerID="e75a988ab6af2037fbe54854abce6c02d2d060586dce547ea9342e6c86efb7ad"
Oct 30 20:30:12.915280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.914825    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"]
Oct 30 20:30:12.929497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.923538    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-843ea8-29889870-2svsx"]
Oct 30 20:30:12.955408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.955377    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"]
Oct 30 20:30:12.957292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:12.957248    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-7975ff79d7-ksht9"]
Oct 30 20:30:13.041702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.041614    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"]
Oct 30 20:30:13.051759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.051723    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd42bdb-5xk6w"]
Oct 30 20:30:13.094309 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.094278    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:13.111163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.111136    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:13.124719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.124692    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cded2d35-8c43-4b09-aa44-24b0bc2600d5-kube-api-access-rfzz6\" (UniqueName: \"kubernetes.io/projected/cded2d35-8c43-4b09-aa44-24b0bc2600d5-kube-api-access-rfzz6\") pod \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") "
Oct 30 20:30:13.124863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.124731    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-openshift-ca\") pod \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") "
Oct 30 20:30:13.124921 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.124899    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-xks-ca\") pod \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") "
Oct 30 20:30:13.125052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.125034    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-xks-ca\") pod \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") "
Oct 30 20:30:13.125231 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.125210    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-openshift-ca\") pod \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\" (UID: \"cded2d35-8c43-4b09-aa44-24b0bc2600d5\") "
Oct 30 20:30:13.125291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.125274    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-kube-api-access-jrkvs\" (UniqueName: \"kubernetes.io/projected/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-kube-api-access-jrkvs\") pod \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\" (UID: \"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d\") "
Oct 30 20:30:13.125354 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:13.125038    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480db7f0_53bf_4af2_8ad2_f7d92fc138ab.slice/crio-748dcfc233234a77fa3226267056b11a62cc9351b3b57022fcfb5aaae450dc8f WatchSource:0}: Error finding container 748dcfc233234a77fa3226267056b11a62cc9351b3b57022fcfb5aaae450dc8f: Status 404 returned error can't find the container with id 748dcfc233234a77fa3226267056b11a62cc9351b3b57022fcfb5aaae450dc8f
Oct 30 20:30:13.125408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.125111    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-openshift-ca" pod "617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" (UID: "617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:13.125711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.125694    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.125783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.125765    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-openshift-ca" pod "cded2d35-8c43-4b09-aa44-24b0bc2600d5" (UID: "cded2d35-8c43-4b09-aa44-24b0bc2600d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:13.126610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.126587    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"]
Oct 30 20:30:13.127428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.127398    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cded2d35-8c43-4b09-aa44-24b0bc2600d5-kube-api-access-rfzz6" pod "cded2d35-8c43-4b09-aa44-24b0bc2600d5" (UID: "cded2d35-8c43-4b09-aa44-24b0bc2600d5"). InnerVolumeSpecName "kube-api-access-rfzz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:13.127756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.127732    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-xks-ca" pod "cded2d35-8c43-4b09-aa44-24b0bc2600d5" (UID: "cded2d35-8c43-4b09-aa44-24b0bc2600d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:13.129189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.129170    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-kube-api-access-jrkvs" pod "617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" (UID: "617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d"). InnerVolumeSpecName "kube-api-access-jrkvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:13.129814 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.129792    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-xks-ca" pod "617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" (UID: "617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:13.227032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.227000    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.227032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.227029    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.227032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.227039    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cded2d35-8c43-4b09-aa44-24b0bc2600d5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.227271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.227049    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrkvs\" (UniqueName: \"kubernetes.io/projected/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d-kube-api-access-jrkvs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.227271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.227058    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfzz6\" (UniqueName: \"kubernetes.io/projected/cded2d35-8c43-4b09-aa44-24b0bc2600d5-kube-api-access-rfzz6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.352065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.351973    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main" containerID="cri-o://7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812" gracePeriod=45
Oct 30 20:30:13.629636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.629596    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aade8fd-6341-43f8-9b0e-86353874976a" path="/var/lib/kubelet/pods/0aade8fd-6341-43f8-9b0e-86353874976a/volumes"
Oct 30 20:30:13.630289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.630262    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e" path="/var/lib/kubelet/pods/13ba0576-c136-4bff-ba0d-925da9b0488e/volumes"
Oct 30 20:30:13.630881 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.630851    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f594f2c-af03-4c8d-98c5-4d3ae876650a" path="/var/lib/kubelet/pods/6f594f2c-af03-4c8d-98c5-4d3ae876650a/volumes"
Oct 30 20:30:13.631952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.631927    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc93ebb-ad6d-4cae-8d00-036fce3e529b" path="/var/lib/kubelet/pods/9bc93ebb-ad6d-4cae-8d00-036fce3e529b/volumes"
Oct 30 20:30:13.641552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.641525    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 20:30:13.734395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.734362    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tmp-dir\") pod \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") "
Oct 30 20:30:13.734641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.734410    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-home\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-home\") pod \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") "
Oct 30 20:30:13.734641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.734442    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-dshm\") pod \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") "
Oct 30 20:30:13.734641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.734578    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tls-certs\") pod \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") "
Oct 30 20:30:13.734641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.734623    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-model-cache\") pod \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") "
Oct 30 20:30:13.734892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.734671    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-kube-api-access-hcmmf\" (UniqueName: \"kubernetes.io/projected/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-kube-api-access-hcmmf\") pod \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\" (UID: \"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9\") "
Oct 30 20:30:13.734983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.734942    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tmp-dir" pod "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" (UID: "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:13.735317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.735293    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.735428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.735395    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-model-cache" pod "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" (UID: "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:13.735428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.735412    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-home" pod "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" (UID: "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:13.737371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.737331    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-kube-api-access-hcmmf" pod "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" (UID: "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9"). InnerVolumeSpecName "kube-api-access-hcmmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:13.737653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.737620    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-dshm" pod "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" (UID: "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:13.737776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.737756    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tls-certs" pod "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" (UID: "ef7e3f0b-4ff8-4a13-a928-8b99655a00e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:13.829127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.829096    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"
Oct 30 20:30:13.829590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.829132    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6" event={"ID":"617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d","Type":"ContainerDied","Data":"4635ef2de8d8065416556bdf7a92940c7122d0f323b20e2ce9545737f3c93ee9"}
Oct 30 20:30:13.829590 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.829190    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4635ef2de8d8065416556bdf7a92940c7122d0f323b20e2ce9545737f3c93ee9"
Oct 30 20:30:13.830763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.830737    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"
Oct 30 20:30:13.830763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.830747    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p" event={"ID":"cded2d35-8c43-4b09-aa44-24b0bc2600d5","Type":"ContainerDied","Data":"73ab3355a6f446ea46ade381f2e70b6bbc349cf3349891ef78a1d2f0e3b5a00b"}
Oct 30 20:30:13.830932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.830784    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ab3355a6f446ea46ade381f2e70b6bbc349cf3349891ef78a1d2f0e3b5a00b"
Oct 30 20:30:13.833338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.832976    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf" event={"ID":"480db7f0-53bf-4af2-8ad2-f7d92fc138ab","Type":"ContainerStarted","Data":"e0c96eccfbb2d4623c30a2dd803b961f1d949418e747f67b8b9f37e4081cc047"}
Oct 30 20:30:13.833338 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.833027    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf" event={"ID":"480db7f0-53bf-4af2-8ad2-f7d92fc138ab","Type":"ContainerStarted","Data":"748dcfc233234a77fa3226267056b11a62cc9351b3b57022fcfb5aaae450dc8f"}
Oct 30 20:30:13.834579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.834558    4519 generic.go:356] "Generic (PLEG): container finished" podID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerID="7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812" exitCode=0
Oct 30 20:30:13.834689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.834591    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"
Oct 30 20:30:13.834689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.834638    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" event={"ID":"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9","Type":"ContainerDied","Data":"7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812"}
Oct 30 20:30:13.834689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.834683    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b" event={"ID":"ef7e3f0b-4ff8-4a13-a928-8b99655a00e9","Type":"ContainerDied","Data":"69cfd2f10edc334a89a2fcafa193533432ea42d48d42b5f1d37cc733b7626bc8"}
Oct 30 20:30:13.834838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.834706    4519 scope.go:122] "RemoveContainer" containerID="7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812"
Oct 30 20:30:13.836099 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.836079    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.836181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.836103    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.836181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.836116    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.836181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.836130    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.836181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.836144    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcmmf\" (UniqueName: \"kubernetes.io/projected/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9-kube-api-access-hcmmf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:13.845717 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.845702    4519 scope.go:122] "RemoveContainer" containerID="7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812"
Oct 30 20:30:13.845938 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:13.845918    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812\": container with ID starting with 7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812 not found: ID does not exist" containerID="7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812"
Oct 30 20:30:13.845978 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.845947    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812"} err="failed to get container status \"7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812\": rpc error: code = NotFound desc = could not find container \"7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812\": container with ID starting with 7fe1089eeb57aa1842854e4b1b6d887ff53c852687a8efdd7dfb0c7a9f5a5812 not found: ID does not exist"
Oct 30 20:30:13.880372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.880266    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf" podStartSLOduration=1.880253122 podStartE2EDuration="1.880253122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:13.878535753 +0000 UTC m=+3338.379239477" watchObservedRunningTime="2026-10-30 20:30:13.880253122 +0000 UTC m=+3338.380956845"
Oct 30 20:30:13.924614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.924565    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"]
Oct 30 20:30:13.929836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:13.929804    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/simulator-w3-01f7d3-kserve-6d4d67dbdf-k5w9b"]
Oct 30 20:30:14.849043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:14.849002    4519 generic.go:356] "Generic (PLEG): container finished" podID="480db7f0-53bf-4af2-8ad2-f7d92fc138ab" containerID="e0c96eccfbb2d4623c30a2dd803b961f1d949418e747f67b8b9f37e4081cc047" exitCode=0
Oct 30 20:30:14.849552 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:14.849088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf" event={"ID":"480db7f0-53bf-4af2-8ad2-f7d92fc138ab","Type":"ContainerDied","Data":"e0c96eccfbb2d4623c30a2dd803b961f1d949418e747f67b8b9f37e4081cc047"}
Oct 30 20:30:15.122957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:15.122874    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"]
Oct 30 20:30:15.144188 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:15.144147    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d130d9ef-rplcf"]
Oct 30 20:30:15.612721 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:15.612684    4519 scope.go:122] "RemoveContainer" containerID="93ead50f68ed083fd7c0943d8dacc514f5fa887ffb78e7e227e62c4eda5599f8"
Oct 30 20:30:15.612938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:15.612826    4519 scope.go:122] "RemoveContainer" containerID="5223f4139d4d689160b3b581c7ae36be118e846924d6480172d1cb412476a300"
Oct 30 20:30:15.619141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:15.619112    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c4fb3d-0052-422f-86db-92f5a57fac28" path="/var/lib/kubelet/pods/a3c4fb3d-0052-422f-86db-92f5a57fac28/volumes"
Oct 30 20:30:15.619765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:15.619738    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" path="/var/lib/kubelet/pods/ef7e3f0b-4ff8-4a13-a928-8b99655a00e9/volumes"
Oct 30 20:30:16.219258 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.219222    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:16.270714 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.270680    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-openshift-ca\") pod \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") "
Oct 30 20:30:16.271615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.270922    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-xks-ca\") pod \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") "
Oct 30 20:30:16.271615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.270976    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-kube-api-access-7x2wd\" (UniqueName: \"kubernetes.io/projected/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-kube-api-access-7x2wd\") pod \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\" (UID: \"480db7f0-53bf-4af2-8ad2-f7d92fc138ab\") "
Oct 30 20:30:16.271615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.271547    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-openshift-ca" pod "480db7f0-53bf-4af2-8ad2-f7d92fc138ab" (UID: "480db7f0-53bf-4af2-8ad2-f7d92fc138ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:16.274295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.274270    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-xks-ca" pod "480db7f0-53bf-4af2-8ad2-f7d92fc138ab" (UID: "480db7f0-53bf-4af2-8ad2-f7d92fc138ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:16.274807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.274783    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-kube-api-access-7x2wd" pod "480db7f0-53bf-4af2-8ad2-f7d92fc138ab" (UID: "480db7f0-53bf-4af2-8ad2-f7d92fc138ab"). InnerVolumeSpecName "kube-api-access-7x2wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:16.372197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.372160    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x2wd\" (UniqueName: \"kubernetes.io/projected/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-kube-api-access-7x2wd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:16.372197 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.372197    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:16.372414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.372211    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480db7f0-53bf-4af2-8ad2-f7d92fc138ab-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:16.605319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.605243    4519 scope.go:122] "RemoveContainer" containerID="7e7a601d8c040a9e60269cb960d851dcf1a6c682a72a49a4327401299c712832"
Oct 30 20:30:16.872303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.872271    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"
Oct 30 20:30:16.872303 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.872286    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf" event={"ID":"480db7f0-53bf-4af2-8ad2-f7d92fc138ab","Type":"ContainerDied","Data":"748dcfc233234a77fa3226267056b11a62cc9351b3b57022fcfb5aaae450dc8f"}
Oct 30 20:30:16.872519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.872335    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748dcfc233234a77fa3226267056b11a62cc9351b3b57022fcfb5aaae450dc8f"
Oct 30 20:30:16.874853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.874822    4519 generic.go:356] "Generic (PLEG): container finished" podID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerID="d2d2eaeb5c5dfd57822cb7bc823e23a8aacefae9c460b8489f6639c3d5837fca" exitCode=60
Oct 30 20:30:16.874969 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.874903    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" event={"ID":"bcbcab0c-a105-4dd1-a417-f324ba5bf305","Type":"ContainerDied","Data":"d2d2eaeb5c5dfd57822cb7bc823e23a8aacefae9c460b8489f6639c3d5837fca"}
Oct 30 20:30:16.874969 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.874960    4519 scope.go:122] "RemoveContainer" containerID="5223f4139d4d689160b3b581c7ae36be118e846924d6480172d1cb412476a300"
Oct 30 20:30:16.875569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.875545    4519 scope.go:122] "RemoveContainer" containerID="d2d2eaeb5c5dfd57822cb7bc823e23a8aacefae9c460b8489f6639c3d5837fca"
Oct 30 20:30:16.875875 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:16.875848    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg_odh-ai-gateway-infra(bcbcab0c-a105-4dd1-a417-f324ba5bf305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305"
Oct 30 20:30:16.877518 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.877495    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerStarted","Data":"cf9da1dab79657b3eecf8e4a181abf70e7cdda9c70e45b7219be084b2f1592a5"}
Oct 30 20:30:16.880040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.879992    4519 generic.go:356] "Generic (PLEG): container finished" podID="42663d1a-4550-41bb-920b-52fdeb79efee" containerID="683ad5e7399978f34e85da2011637ee3ee4c5ee4020d5379d87150069f665226" exitCode=60
Oct 30 20:30:16.880040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.880016    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" event={"ID":"42663d1a-4550-41bb-920b-52fdeb79efee","Type":"ContainerDied","Data":"683ad5e7399978f34e85da2011637ee3ee4c5ee4020d5379d87150069f665226"}
Oct 30 20:30:16.880391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.880371    4519 scope.go:122] "RemoveContainer" containerID="683ad5e7399978f34e85da2011637ee3ee4c5ee4020d5379d87150069f665226"
Oct 30 20:30:16.880681 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:16.880628    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9_odh-ai-gateway-infra(42663d1a-4550-41bb-920b-52fdeb79efee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" podUID="42663d1a-4550-41bb-920b-52fdeb79efee"
Oct 30 20:30:16.892974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:16.892952    4519 scope.go:122] "RemoveContainer" containerID="93ead50f68ed083fd7c0943d8dacc514f5fa887ffb78e7e227e62c4eda5599f8"
Oct 30 20:30:17.606012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.605979    4519 scope.go:122] "RemoveContainer" containerID="e8d10558f7030b49e06564d2bc7c396c2a5a1e897d3c70aaaf11223990d068fc"
Oct 30 20:30:17.606506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.606194    4519 scope.go:122] "RemoveContainer" containerID="f2f9438291e8e399f9c5738049e94dca42f170654bfbe21d3e23a79fb019a186"
Oct 30 20:30:17.606506 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.606377    4519 scope.go:122] "RemoveContainer" containerID="f4481bd7e5a6059f2b8e92c5b2d2544341353cbb46280215ac1f7b7da7e47a4f"
Oct 30 20:30:17.901413 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.901375    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"]
Oct 30 20:30:17.909045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.909007    4519 generic.go:356] "Generic (PLEG): container finished" podID="02c434ed-d6fa-4857-9642-f0627666212e" containerID="cf9da1dab79657b3eecf8e4a181abf70e7cdda9c70e45b7219be084b2f1592a5" exitCode=60
Oct 30 20:30:17.909225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.909205    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerDied","Data":"cf9da1dab79657b3eecf8e4a181abf70e7cdda9c70e45b7219be084b2f1592a5"}
Oct 30 20:30:17.909324 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.909269    4519 scope.go:122] "RemoveContainer" containerID="7e7a601d8c040a9e60269cb960d851dcf1a6c682a72a49a4327401299c712832"
Oct 30 20:30:17.909760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.909741    4519 scope.go:122] "RemoveContainer" containerID="cf9da1dab79657b3eecf8e4a181abf70e7cdda9c70e45b7219be084b2f1592a5"
Oct 30 20:30:17.910103 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:17.910077    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-b5rg2_odh-ai-gateway-infra(02c434ed-d6fa-4857-9642-f0627666212e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" podUID="02c434ed-d6fa-4857-9642-f0627666212e"
Oct 30 20:30:17.966519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:17.966096    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"]
Oct 30 20:30:18.052698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.052372    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"]
Oct 30 20:30:18.125719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.125652    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"]
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.240083    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"]
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241010    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6f594f2c-af03-4c8d-98c5-4d3ae876650a" containerName="payload-pre-processing"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241034    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f594f2c-af03-4c8d-98c5-4d3ae876650a" containerName="payload-pre-processing"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241062    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241070    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241094    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cded2d35-8c43-4b09-aa44-24b0bc2600d5" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241102    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cded2d35-8c43-4b09-aa44-24b0bc2600d5" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241118    4519 cpu_manager.go:414] "containerMap: removing container" podUID="480db7f0-53bf-4af2-8ad2-f7d92fc138ab" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241127    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480db7f0-53bf-4af2-8ad2-f7d92fc138ab" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241169    4519 cpu_manager.go:414] "containerMap: removing container" podUID="617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241177    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241315    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f594f2c-af03-4c8d-98c5-4d3ae876650a" containerName="payload-pre-processing"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241328    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7e3f0b-4ff8-4a13-a928-8b99655a00e9" containerName="main"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241342    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cded2d35-8c43-4b09-aa44-24b0bc2600d5" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241353    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="480db7f0-53bf-4af2-8ad2-f7d92fc138ab" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241369    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" containerName="revoke-keys"
Oct 30 20:30:18.245206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.241381    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ba0576-c136-4bff-ba0d-925da9b0488e" containerName="cleanup"
Oct 30 20:30:18.258020 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.257980    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.274277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.274154    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:18.282968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.282936    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"]
Oct 30 20:30:18.291660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.289291    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:18.299769 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.299735    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcbcab0c-a105-4dd1-a417-f324ba5bf305-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcab0c-a105-4dd1-a417-f324ba5bf305-maas-api-service-ca\") pod \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") "
Oct 30 20:30:18.299914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.299832    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbcab0c-a105-4dd1-a417-f324ba5bf305-kube-api-access-4njdn\" (UniqueName: \"kubernetes.io/projected/bcbcab0c-a105-4dd1-a417-f324ba5bf305-kube-api-access-4njdn\") pod \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\" (UID: \"bcbcab0c-a105-4dd1-a417-f324ba5bf305\") "
Oct 30 20:30:18.300042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.300011    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.300160 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.300138    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.300248 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.300225    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wrv\" (UniqueName: \"kubernetes.io/projected/fe6107f1-8727-4dc0-b18c-52c2194883ce-kube-api-access-f8wrv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.300793 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.300763    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbcab0c-a105-4dd1-a417-f324ba5bf305-maas-api-service-ca" pod "bcbcab0c-a105-4dd1-a417-f324ba5bf305" (UID: "bcbcab0c-a105-4dd1-a417-f324ba5bf305"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:18.307137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.307092    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbcab0c-a105-4dd1-a417-f324ba5bf305-kube-api-access-4njdn" pod "bcbcab0c-a105-4dd1-a417-f324ba5bf305" (UID: "bcbcab0c-a105-4dd1-a417-f324ba5bf305"). InnerVolumeSpecName "kube-api-access-4njdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:18.401505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.401446    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.401687 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.401662    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.401859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.401759    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wrv\" (UniqueName: \"kubernetes.io/projected/fe6107f1-8727-4dc0-b18c-52c2194883ce-kube-api-access-f8wrv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.402682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.402631    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcab0c-a105-4dd1-a417-f324ba5bf305-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:18.402915 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.402694    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njdn\" (UniqueName: \"kubernetes.io/projected/bcbcab0c-a105-4dd1-a417-f324ba5bf305-kube-api-access-4njdn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:18.403549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.403514    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.404694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.404669    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.421763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.421731    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wrv\" (UniqueName: \"kubernetes.io/projected/fe6107f1-8727-4dc0-b18c-52c2194883ce-kube-api-access-f8wrv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.458448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.458425    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:18.503480 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.503425    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42663d1a-4550-41bb-920b-52fdeb79efee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42663d1a-4550-41bb-920b-52fdeb79efee-maas-api-service-ca\") pod \"42663d1a-4550-41bb-920b-52fdeb79efee\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") "
Oct 30 20:30:18.503979 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.503949    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42663d1a-4550-41bb-920b-52fdeb79efee-maas-api-service-ca" pod "42663d1a-4550-41bb-920b-52fdeb79efee" (UID: "42663d1a-4550-41bb-920b-52fdeb79efee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:18.504300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.504273    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42663d1a-4550-41bb-920b-52fdeb79efee-kube-api-access-7k7dv\" (UniqueName: \"kubernetes.io/projected/42663d1a-4550-41bb-920b-52fdeb79efee-kube-api-access-7k7dv\") pod \"42663d1a-4550-41bb-920b-52fdeb79efee\" (UID: \"42663d1a-4550-41bb-920b-52fdeb79efee\") "
Oct 30 20:30:18.504650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.504630    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42663d1a-4550-41bb-920b-52fdeb79efee-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:18.506936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.506898    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42663d1a-4550-41bb-920b-52fdeb79efee-kube-api-access-7k7dv" pod "42663d1a-4550-41bb-920b-52fdeb79efee" (UID: "42663d1a-4550-41bb-920b-52fdeb79efee"). InnerVolumeSpecName "kube-api-access-7k7dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:18.603174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.603134    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:18.605935 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.605895    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7dv\" (UniqueName: \"kubernetes.io/projected/42663d1a-4550-41bb-920b-52fdeb79efee-kube-api-access-7k7dv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:18.923240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.923210    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"
Oct 30 20:30:18.923751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.923209    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg" event={"ID":"bcbcab0c-a105-4dd1-a417-f324ba5bf305","Type":"ContainerDied","Data":"9d4d303837465953412ff18e7b831904608cc0f6af4ac8486c97b28ae6c2734b"}
Oct 30 20:30:18.923751 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.923393    4519 scope.go:122] "RemoveContainer" containerID="d2d2eaeb5c5dfd57822cb7bc823e23a8aacefae9c460b8489f6639c3d5837fca"
Oct 30 20:30:18.925849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.925810    4519 generic.go:356] "Generic (PLEG): container finished" podID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerID="3c4e0bf2f505937fdac208e166fed5499a87f9014911ee007e1691af29de6901" exitCode=60
Oct 30 20:30:18.925970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.925865    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" event={"ID":"53a0eed0-9ab5-4968-8ba2-09b4b521ab90","Type":"ContainerDied","Data":"3c4e0bf2f505937fdac208e166fed5499a87f9014911ee007e1691af29de6901"}
Oct 30 20:30:18.926237 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.926219    4519 scope.go:122] "RemoveContainer" containerID="3c4e0bf2f505937fdac208e166fed5499a87f9014911ee007e1691af29de6901"
Oct 30 20:30:18.926581 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:18.926553    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx_odh-ai-gateway-infra(53a0eed0-9ab5-4968-8ba2-09b4b521ab90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90"
Oct 30 20:30:18.930432 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.930409    4519 generic.go:356] "Generic (PLEG): container finished" podID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerID="c5bbee72c4818584e8dd502abc567609590460351c59ce746cc98e60c5fd6a1e" exitCode=60
Oct 30 20:30:18.930568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.930499    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" event={"ID":"5ffd813f-fc60-482f-8268-f32c32262e8d","Type":"ContainerDied","Data":"c5bbee72c4818584e8dd502abc567609590460351c59ce746cc98e60c5fd6a1e"}
Oct 30 20:30:18.930833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.930814    4519 scope.go:122] "RemoveContainer" containerID="c5bbee72c4818584e8dd502abc567609590460351c59ce746cc98e60c5fd6a1e"
Oct 30 20:30:18.931139 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:18.931110    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd_odh-ai-gateway-infra(5ffd813f-fc60-482f-8268-f32c32262e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d"
Oct 30 20:30:18.932602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.932583    4519 generic.go:356] "Generic (PLEG): container finished" podID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerID="0330d4c1289f439ef1cd75f4c8f3eda32026c9f62dadb349ad793aa54afca86d" exitCode=60
Oct 30 20:30:18.932685 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.932655    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" event={"ID":"2dd038ac-baa9-4b45-ad00-aef5cdb8127e","Type":"ContainerDied","Data":"0330d4c1289f439ef1cd75f4c8f3eda32026c9f62dadb349ad793aa54afca86d"}
Oct 30 20:30:18.932961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.932947    4519 scope.go:122] "RemoveContainer" containerID="0330d4c1289f439ef1cd75f4c8f3eda32026c9f62dadb349ad793aa54afca86d"
Oct 30 20:30:18.933192 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:18.933175    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x_odh-ai-gateway-infra(2dd038ac-baa9-4b45-ad00-aef5cdb8127e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e"
Oct 30 20:30:18.934086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.934070    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"
Oct 30 20:30:18.934086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.934079    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9" event={"ID":"42663d1a-4550-41bb-920b-52fdeb79efee","Type":"ContainerDied","Data":"c36bbfe7599fbfa51ebd15fd5749d47377e33cf2faead99a7672e7fca3b45aca"}
Oct 30 20:30:18.958589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.958561    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"]
Oct 30 20:30:18.966203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.966175    4519 scope.go:122] "RemoveContainer" containerID="e8d10558f7030b49e06564d2bc7c396c2a5a1e897d3c70aaaf11223990d068fc"
Oct 30 20:30:18.966761 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:18.966728    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6107f1_8727_4dc0_b18c_52c2194883ce.slice/crio-09c76b1a93ece658265c8d8607d2f6d945b913aca6269d8d1597b911e762fcae WatchSource:0}: Error finding container 09c76b1a93ece658265c8d8607d2f6d945b913aca6269d8d1597b911e762fcae: Status 404 returned error can't find the container with id 09c76b1a93ece658265c8d8607d2f6d945b913aca6269d8d1597b911e762fcae
Oct 30 20:30:18.988653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:18.988626    4519 scope.go:122] "RemoveContainer" containerID="f4481bd7e5a6059f2b8e92c5b2d2544341353cbb46280215ac1f7b7da7e47a4f"
Oct 30 20:30:19.011743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.011712    4519 scope.go:122] "RemoveContainer" containerID="f2f9438291e8e399f9c5738049e94dca42f170654bfbe21d3e23a79fb019a186"
Oct 30 20:30:19.033289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.033262    4519 scope.go:122] "RemoveContainer" containerID="683ad5e7399978f34e85da2011637ee3ee4c5ee4020d5379d87150069f665226"
Oct 30 20:30:19.118313 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.118281    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:19.127735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.123216    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c434ed-d6fa-4857-9642-f0627666212e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c434ed-d6fa-4857-9642-f0627666212e-maas-api-service-ca\") pod \"02c434ed-d6fa-4857-9642-f0627666212e\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") "
Oct 30 20:30:19.127735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.123297    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c434ed-d6fa-4857-9642-f0627666212e-kube-api-access-dbs8b\" (UniqueName: \"kubernetes.io/projected/02c434ed-d6fa-4857-9642-f0627666212e-kube-api-access-dbs8b\") pod \"02c434ed-d6fa-4857-9642-f0627666212e\" (UID: \"02c434ed-d6fa-4857-9642-f0627666212e\") "
Oct 30 20:30:19.127735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.124588    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c434ed-d6fa-4857-9642-f0627666212e-maas-api-service-ca" pod "02c434ed-d6fa-4857-9642-f0627666212e" (UID: "02c434ed-d6fa-4857-9642-f0627666212e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:19.130515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.130442    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c434ed-d6fa-4857-9642-f0627666212e-kube-api-access-dbs8b" pod "02c434ed-d6fa-4857-9642-f0627666212e" (UID: "02c434ed-d6fa-4857-9642-f0627666212e"). InnerVolumeSpecName "kube-api-access-dbs8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:19.160363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.159594    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"]
Oct 30 20:30:19.226297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.226207    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c434ed-d6fa-4857-9642-f0627666212e-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:19.226297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.226244    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbs8b\" (UniqueName: \"kubernetes.io/projected/02c434ed-d6fa-4857-9642-f0627666212e-kube-api-access-dbs8b\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:19.376363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.376298    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"]
Oct 30 20:30:19.393353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.392599    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889870-tzgqg"]
Oct 30 20:30:19.455137 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455102    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"]
Oct 30 20:30:19.455676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455659    4519 cpu_manager.go:414] "containerMap: removing container" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:19.455676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455676    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455686    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455691    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455707    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455713    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455729    4519 cpu_manager.go:414] "containerMap: removing container" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455734    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455751    4519 cpu_manager.go:414] "containerMap: removing container" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455755    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455767    4519 cpu_manager.go:414] "containerMap: removing container" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:19.455796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455772    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:19.456093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455865    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:19.456093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455874    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:19.456093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455882    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:19.456093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455889    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:19.456093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455897    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:19.456093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455902    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:19.456093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.455908    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:19.459579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.459551    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.482162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.482082    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"]
Oct 30 20:30:19.521007 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.520963    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"]
Oct 30 20:30:19.526343 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.526314    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5314c2-29889870-p2gd9"]
Oct 30 20:30:19.531381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.531344    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.531520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.531439    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7clj\" (UniqueName: \"kubernetes.io/projected/a195202f-ee30-4ce6-978b-150f09ed08c5-kube-api-access-l7clj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.531643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.531619    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.614395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.614353    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" path="/var/lib/kubelet/pods/42663d1a-4550-41bb-920b-52fdeb79efee/volumes"
Oct 30 20:30:19.617491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.614978    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" path="/var/lib/kubelet/pods/bcbcab0c-a105-4dd1-a417-f324ba5bf305/volumes"
Oct 30 20:30:19.634371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.634345    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.634556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.634375    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7clj\" (UniqueName: \"kubernetes.io/projected/a195202f-ee30-4ce6-978b-150f09ed08c5-kube-api-access-l7clj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.634632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.634615    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.635497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.635429    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.637121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.637096    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.654541 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.654514    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7clj\" (UniqueName: \"kubernetes.io/projected/a195202f-ee30-4ce6-978b-150f09ed08c5-kube-api-access-l7clj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.774945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.774858    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:19.945221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.945192    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"
Oct 30 20:30:19.945221 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.945201    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2" event={"ID":"02c434ed-d6fa-4857-9642-f0627666212e","Type":"ContainerDied","Data":"02cf507ebe0c5a24fe5b38d35a6109b4a7fab20a36d1c431cdfa4b52d2d9d277"}
Oct 30 20:30:19.945710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.945274    4519 scope.go:122] "RemoveContainer" containerID="cf9da1dab79657b3eecf8e4a181abf70e7cdda9c70e45b7219be084b2f1592a5"
Oct 30 20:30:19.954174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.954139    4519 generic.go:356] "Generic (PLEG): container finished" podID="fe6107f1-8727-4dc0-b18c-52c2194883ce" containerID="d0d5d19408c501c99d76e218847312a8736bcf80e67a41913a715b1f4a11356f" exitCode=0
Oct 30 20:30:19.954314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.954245    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9" event={"ID":"fe6107f1-8727-4dc0-b18c-52c2194883ce","Type":"ContainerDied","Data":"d0d5d19408c501c99d76e218847312a8736bcf80e67a41913a715b1f4a11356f"}
Oct 30 20:30:19.954314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:19.954292    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9" event={"ID":"fe6107f1-8727-4dc0-b18c-52c2194883ce","Type":"ContainerStarted","Data":"09c76b1a93ece658265c8d8607d2f6d945b913aca6269d8d1597b911e762fcae"}
Oct 30 20:30:20.037661 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.037577    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"]
Oct 30 20:30:20.041658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.041621    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"]
Oct 30 20:30:20.046372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.046347    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-b5rg2"]
Oct 30 20:30:20.093242 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:20.093185    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda195202f_ee30_4ce6_978b_150f09ed08c5.slice/crio-d7a514201ce644c2060f8944fed9518d5165607b6a9d2cd1d3272ea0d0ad9010 WatchSource:0}: Error finding container d7a514201ce644c2060f8944fed9518d5165607b6a9d2cd1d3272ea0d0ad9010: Status 404 returned error can't find the container with id d7a514201ce644c2060f8944fed9518d5165607b6a9d2cd1d3272ea0d0ad9010
Oct 30 20:30:20.097047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.097011    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"]
Oct 30 20:30:20.124935 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.124877    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"]
Oct 30 20:30:20.199956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.199654    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:20.278046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.278007    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"]
Oct 30 20:30:20.319482 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.319445    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:20.345830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.345781    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffd813f-fc60-482f-8268-f32c32262e8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffd813f-fc60-482f-8268-f32c32262e8d-maas-api-service-ca\") pod \"5ffd813f-fc60-482f-8268-f32c32262e8d\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") "
Oct 30 20:30:20.345970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.345908    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-kube-api-access-cx9n6\" (UniqueName: \"kubernetes.io/projected/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-kube-api-access-cx9n6\") pod \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") "
Oct 30 20:30:20.345970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.345951    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffd813f-fc60-482f-8268-f32c32262e8d-kube-api-access-kvfzf\" (UniqueName: \"kubernetes.io/projected/5ffd813f-fc60-482f-8268-f32c32262e8d-kube-api-access-kvfzf\") pod \"5ffd813f-fc60-482f-8268-f32c32262e8d\" (UID: \"5ffd813f-fc60-482f-8268-f32c32262e8d\") "
Oct 30 20:30:20.346774 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.346741    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-maas-api-service-ca\") pod \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\" (UID: \"2dd038ac-baa9-4b45-ad00-aef5cdb8127e\") "
Oct 30 20:30:20.348037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.347643    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-maas-api-service-ca" pod "2dd038ac-baa9-4b45-ad00-aef5cdb8127e" (UID: "2dd038ac-baa9-4b45-ad00-aef5cdb8127e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:20.348037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.347973    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffd813f-fc60-482f-8268-f32c32262e8d-maas-api-service-ca" pod "5ffd813f-fc60-482f-8268-f32c32262e8d" (UID: "5ffd813f-fc60-482f-8268-f32c32262e8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:20.355610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.355566    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffd813f-fc60-482f-8268-f32c32262e8d-kube-api-access-kvfzf" pod "5ffd813f-fc60-482f-8268-f32c32262e8d" (UID: "5ffd813f-fc60-482f-8268-f32c32262e8d"). InnerVolumeSpecName "kube-api-access-kvfzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:20.357943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.357920    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-kube-api-access-cx9n6" pod "2dd038ac-baa9-4b45-ad00-aef5cdb8127e" (UID: "2dd038ac-baa9-4b45-ad00-aef5cdb8127e"). InnerVolumeSpecName "kube-api-access-cx9n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:20.436968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.436940    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:20.448215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.448186    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-kube-api-access-95t2p\" (UniqueName: \"kubernetes.io/projected/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-kube-api-access-95t2p\") pod \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") "
Oct 30 20:30:20.448342 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.448260    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-maas-api-service-ca\") pod \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\" (UID: \"53a0eed0-9ab5-4968-8ba2-09b4b521ab90\") "
Oct 30 20:30:20.448641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.448621    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx9n6\" (UniqueName: \"kubernetes.io/projected/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-kube-api-access-cx9n6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:20.448641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.448643    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfzf\" (UniqueName: \"kubernetes.io/projected/5ffd813f-fc60-482f-8268-f32c32262e8d-kube-api-access-kvfzf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:20.448806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.448653    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd038ac-baa9-4b45-ad00-aef5cdb8127e-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:20.448806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.448663    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffd813f-fc60-482f-8268-f32c32262e8d-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:20.448806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.448616    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-maas-api-service-ca" pod "53a0eed0-9ab5-4968-8ba2-09b4b521ab90" (UID: "53a0eed0-9ab5-4968-8ba2-09b4b521ab90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:20.450322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.450298    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-kube-api-access-95t2p" pod "53a0eed0-9ab5-4968-8ba2-09b4b521ab90" (UID: "53a0eed0-9ab5-4968-8ba2-09b4b521ab90"). InnerVolumeSpecName "kube-api-access-95t2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:20.549651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.549620    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95t2p\" (UniqueName: \"kubernetes.io/projected/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-kube-api-access-95t2p\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:20.549651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.549648    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a0eed0-9ab5-4968-8ba2-09b4b521ab90-maas-api-service-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:20.964815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.964779    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx" event={"ID":"53a0eed0-9ab5-4968-8ba2-09b4b521ab90","Type":"ContainerDied","Data":"b40f14d5e41a5ff35bccf4663bac2a481a33cc5c1a550608c25fc2b9be8b7e9f"}
Oct 30 20:30:20.965295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.964852    4519 scope.go:122] "RemoveContainer" containerID="3c4e0bf2f505937fdac208e166fed5499a87f9014911ee007e1691af29de6901"
Oct 30 20:30:20.965295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.964791    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"
Oct 30 20:30:20.967229 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.967200    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx" event={"ID":"a195202f-ee30-4ce6-978b-150f09ed08c5","Type":"ContainerStarted","Data":"d23bbbaed24befa91efeb119dba8dd7115ed612e462ced25e864b88c12c077e6"}
Oct 30 20:30:20.967355 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.967252    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx" event={"ID":"a195202f-ee30-4ce6-978b-150f09ed08c5","Type":"ContainerStarted","Data":"d7a514201ce644c2060f8944fed9518d5165607b6a9d2cd1d3272ea0d0ad9010"}
Oct 30 20:30:20.971028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.970993    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd" event={"ID":"5ffd813f-fc60-482f-8268-f32c32262e8d","Type":"ContainerDied","Data":"2c810150c6c0cfeaee7f74a4788ec1332000bdde5dfdd74cd42f715005fcdaf2"}
Oct 30 20:30:20.971165 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.971097    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"
Oct 30 20:30:20.974184 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.974161    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"
Oct 30 20:30:20.974291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.974175    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x" event={"ID":"2dd038ac-baa9-4b45-ad00-aef5cdb8127e","Type":"ContainerDied","Data":"642c9c596a37fdde45965b081f6d2bb2dd11dcd3269f164c1905d31cb6fd1bcb"}
Oct 30 20:30:20.982866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.982752    4519 scope.go:122] "RemoveContainer" containerID="c5bbee72c4818584e8dd502abc567609590460351c59ce746cc98e60c5fd6a1e"
Oct 30 20:30:20.995808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:20.995779    4519 scope.go:122] "RemoveContainer" containerID="0330d4c1289f439ef1cd75f4c8f3eda32026c9f62dadb349ad793aa54afca86d"
Oct 30 20:30:21.021351 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.021323    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"]
Oct 30 20:30:21.039807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.039773    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77ac5a-29889870-8jvhx"]
Oct 30 20:30:21.102317 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.090764    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx" podStartSLOduration=2.09074223 podStartE2EDuration="2.09074223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:21.080860422 +0000 UTC m=+3345.581564146" watchObservedRunningTime="2026-10-30 20:30:21.09074223 +0000 UTC m=+3345.591445955"
Oct 30 20:30:21.111740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.111701    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"]
Oct 30 20:30:21.112033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.111972    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy" containerID="cri-o://4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e" gracePeriod=30
Oct 30 20:30:21.149553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.142070    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"]
Oct 30 20:30:21.149553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.147165    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889870-9kcsd"]
Oct 30 20:30:21.247003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.246912    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"]
Oct 30 20:30:21.261276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.261206    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d2720-29889870-6zh2x"]
Oct 30 20:30:21.275771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.275390    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:21.369134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.369104    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6107f1-8727-4dc0-b18c-52c2194883ce-kube-api-access-f8wrv\" (UniqueName: \"kubernetes.io/projected/fe6107f1-8727-4dc0-b18c-52c2194883ce-kube-api-access-f8wrv\") pod \"fe6107f1-8727-4dc0-b18c-52c2194883ce\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") "
Oct 30 20:30:21.369372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.369306    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-openshift-ca\") pod \"fe6107f1-8727-4dc0-b18c-52c2194883ce\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") "
Oct 30 20:30:21.369372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.369345    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-xks-ca\") pod \"fe6107f1-8727-4dc0-b18c-52c2194883ce\" (UID: \"fe6107f1-8727-4dc0-b18c-52c2194883ce\") "
Oct 30 20:30:21.369764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.369732    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-openshift-ca" pod "fe6107f1-8727-4dc0-b18c-52c2194883ce" (UID: "fe6107f1-8727-4dc0-b18c-52c2194883ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:21.369924 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.369901    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:21.371430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.371399    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6107f1-8727-4dc0-b18c-52c2194883ce-kube-api-access-f8wrv" pod "fe6107f1-8727-4dc0-b18c-52c2194883ce" (UID: "fe6107f1-8727-4dc0-b18c-52c2194883ce"). InnerVolumeSpecName "kube-api-access-f8wrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:21.371548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.371496    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-xks-ca" pod "fe6107f1-8727-4dc0-b18c-52c2194883ce" (UID: "fe6107f1-8727-4dc0-b18c-52c2194883ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:21.471277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.471238    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6107f1-8727-4dc0-b18c-52c2194883ce-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:21.471277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.471276    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8wrv\" (UniqueName: \"kubernetes.io/projected/fe6107f1-8727-4dc0-b18c-52c2194883ce-kube-api-access-f8wrv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:21.613037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.612956    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c434ed-d6fa-4857-9642-f0627666212e" path="/var/lib/kubelet/pods/02c434ed-d6fa-4857-9642-f0627666212e/volumes"
Oct 30 20:30:21.613400 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.613386    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" path="/var/lib/kubelet/pods/2dd038ac-baa9-4b45-ad00-aef5cdb8127e/volumes"
Oct 30 20:30:21.613816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.613802    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" path="/var/lib/kubelet/pods/53a0eed0-9ab5-4968-8ba2-09b4b521ab90/volumes"
Oct 30 20:30:21.614504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.614489    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" path="/var/lib/kubelet/pods/5ffd813f-fc60-482f-8268-f32c32262e8d/volumes"
Oct 30 20:30:21.666842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.666798    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"]
Oct 30 20:30:21.667392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667368    4519 cpu_manager.go:414] "containerMap: removing container" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667393    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667417    4519 cpu_manager.go:414] "containerMap: removing container" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667426    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c434ed-d6fa-4857-9642-f0627666212e" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667447    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667456    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667492    4519 cpu_manager.go:414] "containerMap: removing container" podUID="fe6107f1-8727-4dc0-b18c-52c2194883ce" containerName="revoke-keys"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667502    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6107f1-8727-4dc0-b18c-52c2194883ce" containerName="revoke-keys"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667517    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667524    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667538    4519 cpu_manager.go:414] "containerMap: removing container" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.667549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667545    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667561    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667569    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667590    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667598    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667613    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667620    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667642    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667651    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667662    4519 cpu_manager.go:414] "containerMap: removing container" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667670    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667682    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667691    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667702    4519 cpu_manager.go:414] "containerMap: removing container" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667709    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667865    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6107f1-8727-4dc0-b18c-52c2194883ce" containerName="revoke-keys"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667881    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667891    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667900    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667909    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="42663d1a-4550-41bb-920b-52fdeb79efee" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667921    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667932    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd038ac-baa9-4b45-ad00-aef5cdb8127e" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667941    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667950    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667960    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffd813f-fc60-482f-8268-f32c32262e8d" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667970    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a0eed0-9ab5-4968-8ba2-09b4b521ab90" containerName="cleanup"
Oct 30 20:30:21.668151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.667980    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbcab0c-a105-4dd1-a417-f324ba5bf305" containerName="cleanup"
Oct 30 20:30:21.673383 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.673360    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.691696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.691666    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"]
Oct 30 20:30:21.774553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.774508    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.774746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.774660    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgd4\" (UniqueName: \"kubernetes.io/projected/29fec69b-da51-432c-accd-cd3a4820b487-kube-api-access-tjgd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.774802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.774783    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.875735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.875627    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.875929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.875748    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgd4\" (UniqueName: \"kubernetes.io/projected/29fec69b-da51-432c-accd-cd3a4820b487-kube-api-access-tjgd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.875929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.875877    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.876425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.876397    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.878378 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.878350    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.897991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.897965    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgd4\" (UniqueName: \"kubernetes.io/projected/29fec69b-da51-432c-accd-cd3a4820b487-kube-api-access-tjgd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.983631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.983603    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"
Oct 30 20:30:21.984096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.983640    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9" event={"ID":"fe6107f1-8727-4dc0-b18c-52c2194883ce","Type":"ContainerDied","Data":"09c76b1a93ece658265c8d8607d2f6d945b913aca6269d8d1597b911e762fcae"}
Oct 30 20:30:21.984096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.983734    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09c76b1a93ece658265c8d8607d2f6d945b913aca6269d8d1597b911e762fcae"
Oct 30 20:30:21.986682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.986654    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:21.987141 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.987112    4519 generic.go:356] "Generic (PLEG): container finished" podID="a195202f-ee30-4ce6-978b-150f09ed08c5" containerID="d23bbbaed24befa91efeb119dba8dd7115ed612e462ced25e864b88c12c077e6" exitCode=0
Oct 30 20:30:21.987264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:21.987214    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx" event={"ID":"a195202f-ee30-4ce6-978b-150f09ed08c5","Type":"ContainerDied","Data":"d23bbbaed24befa91efeb119dba8dd7115ed612e462ced25e864b88c12c077e6"}
Oct 30 20:30:22.299363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:22.299333    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"]
Oct 30 20:30:22.301738 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:22.301699    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fec69b_da51_432c_accd_cd3a4820b487.slice/crio-d60ec0ec464e05305f81f4277ee6dde310fa515fd22f079ad0e4e9d67446b89e WatchSource:0}: Error finding container d60ec0ec464e05305f81f4277ee6dde310fa515fd22f079ad0e4e9d67446b89e: Status 404 returned error can't find the container with id d60ec0ec464e05305f81f4277ee6dde310fa515fd22f079ad0e4e9d67446b89e
Oct 30 20:30:23.000095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.000043    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq" event={"ID":"29fec69b-da51-432c-accd-cd3a4820b487","Type":"ContainerStarted","Data":"3e573943e2ba8878e77715ff218b4eb226fd9e05cff5467662bed3283fad4de8"}
Oct 30 20:30:23.000591 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.000121    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq" event={"ID":"29fec69b-da51-432c-accd-cd3a4820b487","Type":"ContainerStarted","Data":"d60ec0ec464e05305f81f4277ee6dde310fa515fd22f079ad0e4e9d67446b89e"}
Oct 30 20:30:23.044505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.041771    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq" podStartSLOduration=2.041751672 podStartE2EDuration="2.041751672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:23.041279454 +0000 UTC m=+3347.541983179" watchObservedRunningTime="2026-10-30 20:30:23.041751672 +0000 UTC m=+3347.542455395"
Oct 30 20:30:23.233530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.233503    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:23.294527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.294396    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-openshift-ca\") pod \"a195202f-ee30-4ce6-978b-150f09ed08c5\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") "
Oct 30 20:30:23.294692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.294612    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-xks-ca\") pod \"a195202f-ee30-4ce6-978b-150f09ed08c5\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") "
Oct 30 20:30:23.294830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.294750    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a195202f-ee30-4ce6-978b-150f09ed08c5-kube-api-access-l7clj\" (UniqueName: \"kubernetes.io/projected/a195202f-ee30-4ce6-978b-150f09ed08c5-kube-api-access-l7clj\") pod \"a195202f-ee30-4ce6-978b-150f09ed08c5\" (UID: \"a195202f-ee30-4ce6-978b-150f09ed08c5\") "
Oct 30 20:30:23.294830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.294782    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-openshift-ca" pod "a195202f-ee30-4ce6-978b-150f09ed08c5" (UID: "a195202f-ee30-4ce6-978b-150f09ed08c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:23.295273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.295246    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.297004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.296974    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a195202f-ee30-4ce6-978b-150f09ed08c5-kube-api-access-l7clj" pod "a195202f-ee30-4ce6-978b-150f09ed08c5" (UID: "a195202f-ee30-4ce6-978b-150f09ed08c5"). InnerVolumeSpecName "kube-api-access-l7clj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:23.297363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.297335    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-xks-ca" pod "a195202f-ee30-4ce6-978b-150f09ed08c5" (UID: "a195202f-ee30-4ce6-978b-150f09ed08c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:23.347501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.347406    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8" podUID="efe0516c-11f3-4c15-bc16-a60322bdbcbe" containerName="main" containerID="cri-o://e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af" gracePeriod=45
Oct 30 20:30:23.396957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.396926    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7clj\" (UniqueName: \"kubernetes.io/projected/a195202f-ee30-4ce6-978b-150f09ed08c5-kube-api-access-l7clj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.396957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.396960    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a195202f-ee30-4ce6-978b-150f09ed08c5-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.609255 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.609230    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:30:23.700760 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.700719    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe0516c-11f3-4c15-bc16-a60322bdbcbe-kube-api-access-7tjd6\" (UniqueName: \"kubernetes.io/projected/efe0516c-11f3-4c15-bc16-a60322bdbcbe-kube-api-access-7tjd6\") pod \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") "
Oct 30 20:30:23.700961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.700842    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-home\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-home\") pod \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") "
Oct 30 20:30:23.700961 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.700931    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tmp-dir\") pod \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") "
Oct 30 20:30:23.701099 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.700971    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-model-cache\") pod \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") "
Oct 30 20:30:23.701099 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701005    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tls-certs\") pod \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") "
Oct 30 20:30:23.701099 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701038    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-dshm\") pod \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\" (UID: \"efe0516c-11f3-4c15-bc16-a60322bdbcbe\") "
Oct 30 20:30:23.701276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701141    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-home" pod "efe0516c-11f3-4c15-bc16-a60322bdbcbe" (UID: "efe0516c-11f3-4c15-bc16-a60322bdbcbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:23.701276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701226    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tmp-dir" pod "efe0516c-11f3-4c15-bc16-a60322bdbcbe" (UID: "efe0516c-11f3-4c15-bc16-a60322bdbcbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:23.701403 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701309    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-model-cache" pod "efe0516c-11f3-4c15-bc16-a60322bdbcbe" (UID: "efe0516c-11f3-4c15-bc16-a60322bdbcbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:23.701660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701635    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.701660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701662    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.701849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.701676    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.703563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.703534    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tls-certs" pod "efe0516c-11f3-4c15-bc16-a60322bdbcbe" (UID: "efe0516c-11f3-4c15-bc16-a60322bdbcbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:23.703672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.703594    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-dshm" pod "efe0516c-11f3-4c15-bc16-a60322bdbcbe" (UID: "efe0516c-11f3-4c15-bc16-a60322bdbcbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:23.703759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.703742    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe0516c-11f3-4c15-bc16-a60322bdbcbe-kube-api-access-7tjd6" pod "efe0516c-11f3-4c15-bc16-a60322bdbcbe" (UID: "efe0516c-11f3-4c15-bc16-a60322bdbcbe"). InnerVolumeSpecName "kube-api-access-7tjd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:23.802963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.802927    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tjd6\" (UniqueName: \"kubernetes.io/projected/efe0516c-11f3-4c15-bc16-a60322bdbcbe-kube-api-access-7tjd6\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.803166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.802965    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe0516c-11f3-4c15-bc16-a60322bdbcbe-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.803166 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.802987    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe0516c-11f3-4c15-bc16-a60322bdbcbe-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:23.922071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:23.922033    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8" podUID="c6854c09-8a7c-4668-844d-7ce7ac766060" containerName="main" containerID="cri-o://983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d" gracePeriod=45
Oct 30 20:30:24.009439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.009398    4519 generic.go:356] "Generic (PLEG): container finished" podID="efe0516c-11f3-4c15-bc16-a60322bdbcbe" containerID="e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af" exitCode=0
Oct 30 20:30:24.009932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.009447    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"
Oct 30 20:30:24.009932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.009507    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8" event={"ID":"efe0516c-11f3-4c15-bc16-a60322bdbcbe","Type":"ContainerDied","Data":"e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af"}
Oct 30 20:30:24.009932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.009551    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8" event={"ID":"efe0516c-11f3-4c15-bc16-a60322bdbcbe","Type":"ContainerDied","Data":"a741e69c3c49a650b7ee93fb56edd9bf10f107ae211d85cfe3d9e8af5efb2085"}
Oct 30 20:30:24.009932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.009575    4519 scope.go:122] "RemoveContainer" containerID="e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af"
Oct 30 20:30:24.011330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.011310    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"
Oct 30 20:30:24.011458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.011332    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx" event={"ID":"a195202f-ee30-4ce6-978b-150f09ed08c5","Type":"ContainerDied","Data":"d7a514201ce644c2060f8944fed9518d5165607b6a9d2cd1d3272ea0d0ad9010"}
Oct 30 20:30:24.011458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.011383    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7a514201ce644c2060f8944fed9518d5165607b6a9d2cd1d3272ea0d0ad9010"
Oct 30 20:30:24.013574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.013428    4519 generic.go:356] "Generic (PLEG): container finished" podID="29fec69b-da51-432c-accd-cd3a4820b487" containerID="3e573943e2ba8878e77715ff218b4eb226fd9e05cff5467662bed3283fad4de8" exitCode=0
Oct 30 20:30:24.013574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.013523    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq" event={"ID":"29fec69b-da51-432c-accd-cd3a4820b487","Type":"ContainerDied","Data":"3e573943e2ba8878e77715ff218b4eb226fd9e05cff5467662bed3283fad4de8"}
Oct 30 20:30:24.042449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.042423    4519 scope.go:122] "RemoveContainer" containerID="e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af"
Oct 30 20:30:24.042902 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:24.042880    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af\": container with ID starting with e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af not found: ID does not exist" containerID="e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af"
Oct 30 20:30:24.043017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.042909    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af"} err="failed to get container status \"e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af\": rpc error: code = NotFound desc = could not find container \"e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af\": container with ID starting with e974044c6c62515fe8fb09799a8f50728ecbe7c1a424716e913b8efcc9a722af not found: ID does not exist"
Oct 30 20:30:24.065564 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.065534    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"]
Oct 30 20:30:24.077181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.077152    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/rate-test-model-77ac5a-kserve-77dbc55885-2zvq8"]
Oct 30 20:30:24.184050 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.184006    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"]
Oct 30 20:30:24.188316 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.188274    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy" containerID="cri-o://35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c" gracePeriod=30
Oct 30 20:30:24.231534 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.231508    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:30:24.307768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.307732    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-model-cache\") pod \"c6854c09-8a7c-4668-844d-7ce7ac766060\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") "
Oct 30 20:30:24.307985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.307915    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-home\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-home\") pod \"c6854c09-8a7c-4668-844d-7ce7ac766060\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") "
Oct 30 20:30:24.307985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.307976    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6854c09-8a7c-4668-844d-7ce7ac766060-kube-api-access-tm2gl\" (UniqueName: \"kubernetes.io/projected/c6854c09-8a7c-4668-844d-7ce7ac766060-kube-api-access-tm2gl\") pod \"c6854c09-8a7c-4668-844d-7ce7ac766060\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") "
Oct 30 20:30:24.308133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.308005    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-dshm\") pod \"c6854c09-8a7c-4668-844d-7ce7ac766060\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") "
Oct 30 20:30:24.308133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.308040    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-tmp-dir\") pod \"c6854c09-8a7c-4668-844d-7ce7ac766060\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") "
Oct 30 20:30:24.308133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.308066    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-model-cache" pod "c6854c09-8a7c-4668-844d-7ce7ac766060" (UID: "c6854c09-8a7c-4668-844d-7ce7ac766060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:24.308133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.308078    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6854c09-8a7c-4668-844d-7ce7ac766060-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6854c09-8a7c-4668-844d-7ce7ac766060-tls-certs\") pod \"c6854c09-8a7c-4668-844d-7ce7ac766060\" (UID: \"c6854c09-8a7c-4668-844d-7ce7ac766060\") "
Oct 30 20:30:24.308371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.308320    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-home" pod "c6854c09-8a7c-4668-844d-7ce7ac766060" (UID: "c6854c09-8a7c-4668-844d-7ce7ac766060"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:24.308554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.308526    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-tmp-dir" pod "c6854c09-8a7c-4668-844d-7ce7ac766060" (UID: "c6854c09-8a7c-4668-844d-7ce7ac766060"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:24.309032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.309001    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:24.309032 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.309032    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:24.309228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.309071    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:24.311148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.311118    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-dshm" pod "c6854c09-8a7c-4668-844d-7ce7ac766060" (UID: "c6854c09-8a7c-4668-844d-7ce7ac766060"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:24.311593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.311564    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6854c09-8a7c-4668-844d-7ce7ac766060-tls-certs" pod "c6854c09-8a7c-4668-844d-7ce7ac766060" (UID: "c6854c09-8a7c-4668-844d-7ce7ac766060"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:24.313630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.313604    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6854c09-8a7c-4668-844d-7ce7ac766060-kube-api-access-tm2gl" pod "c6854c09-8a7c-4668-844d-7ce7ac766060" (UID: "c6854c09-8a7c-4668-844d-7ce7ac766060"). InnerVolumeSpecName "kube-api-access-tm2gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:24.410289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.410257    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm2gl\" (UniqueName: \"kubernetes.io/projected/c6854c09-8a7c-4668-844d-7ce7ac766060-kube-api-access-tm2gl\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:24.410289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.410286    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6854c09-8a7c-4668-844d-7ce7ac766060-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:24.410289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:24.410297    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6854c09-8a7c-4668-844d-7ce7ac766060-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:25.024065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.024023    4519 generic.go:356] "Generic (PLEG): container finished" podID="c6854c09-8a7c-4668-844d-7ce7ac766060" containerID="983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d" exitCode=0
Oct 30 20:30:25.024626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.024073    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"
Oct 30 20:30:25.024626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.024112    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8" event={"ID":"c6854c09-8a7c-4668-844d-7ce7ac766060","Type":"ContainerDied","Data":"983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d"}
Oct 30 20:30:25.024626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.024151    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8" event={"ID":"c6854c09-8a7c-4668-844d-7ce7ac766060","Type":"ContainerDied","Data":"2646808a89daf698012a6af7b2c2920ac4f83a43af59aa46493dd4995c9739a6"}
Oct 30 20:30:25.024626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.024174    4519 scope.go:122] "RemoveContainer" containerID="983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d"
Oct 30 20:30:25.036429 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.036406    4519 scope.go:122] "RemoveContainer" containerID="983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d"
Oct 30 20:30:25.036783 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:25.036755    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d\": container with ID starting with 983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d not found: ID does not exist" containerID="983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d"
Oct 30 20:30:25.036895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.036796    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d"} err="failed to get container status \"983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d\": rpc error: code = NotFound desc = could not find container \"983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d\": container with ID starting with 983cacc21d2e1f0747463e91ae647f5618906cc1cc33635577c2c290f6f6122d not found: ID does not exist"
Oct 30 20:30:25.091680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.083706    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"]
Oct 30 20:30:25.098389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.098352    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/rate-test-model-5314c2-kserve-66bb55ff97-vflj8"]
Oct 30 20:30:25.192980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.192954    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:25.220302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.220272    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fec69b-da51-432c-accd-cd3a4820b487-kube-api-access-tjgd4\" (UniqueName: \"kubernetes.io/projected/29fec69b-da51-432c-accd-cd3a4820b487-kube-api-access-tjgd4\") pod \"29fec69b-da51-432c-accd-cd3a4820b487\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") "
Oct 30 20:30:25.220493 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.220366    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-openshift-ca\") pod \"29fec69b-da51-432c-accd-cd3a4820b487\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") "
Oct 30 20:30:25.220587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.220522    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-xks-ca\") pod \"29fec69b-da51-432c-accd-cd3a4820b487\" (UID: \"29fec69b-da51-432c-accd-cd3a4820b487\") "
Oct 30 20:30:25.220749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.220721    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-openshift-ca" pod "29fec69b-da51-432c-accd-cd3a4820b487" (UID: "29fec69b-da51-432c-accd-cd3a4820b487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:25.221036 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.220976    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:25.222611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.222587    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-xks-ca" pod "29fec69b-da51-432c-accd-cd3a4820b487" (UID: "29fec69b-da51-432c-accd-cd3a4820b487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:25.222693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.222642    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fec69b-da51-432c-accd-cd3a4820b487-kube-api-access-tjgd4" pod "29fec69b-da51-432c-accd-cd3a4820b487" (UID: "29fec69b-da51-432c-accd-cd3a4820b487"). InnerVolumeSpecName "kube-api-access-tjgd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:25.322354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.322275    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fec69b-da51-432c-accd-cd3a4820b487-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:25.322354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.322309    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgd4\" (UniqueName: \"kubernetes.io/projected/29fec69b-da51-432c-accd-cd3a4820b487-kube-api-access-tjgd4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:25.613205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.613119    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6854c09-8a7c-4668-844d-7ce7ac766060" path="/var/lib/kubelet/pods/c6854c09-8a7c-4668-844d-7ce7ac766060/volumes"
Oct 30 20:30:25.613633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:25.613617    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe0516c-11f3-4c15-bc16-a60322bdbcbe" path="/var/lib/kubelet/pods/efe0516c-11f3-4c15-bc16-a60322bdbcbe/volumes"
Oct 30 20:30:26.034498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.034433    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"
Oct 30 20:30:26.034498 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.034457    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq" event={"ID":"29fec69b-da51-432c-accd-cd3a4820b487","Type":"ContainerDied","Data":"d60ec0ec464e05305f81f4277ee6dde310fa515fd22f079ad0e4e9d67446b89e"}
Oct 30 20:30:26.034946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.034518    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60ec0ec464e05305f81f4277ee6dde310fa515fd22f079ad0e4e9d67446b89e"
Oct 30 20:30:26.402550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.402516    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:30:26.434945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.434912    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29106e39-77cd-4a3c-8766-984760dc28ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29106e39-77cd-4a3c-8766-984760dc28ad-istio-podinfo\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.434976    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-kube-api-access-r7pld\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-kube-api-access-r7pld\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435040    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29106e39-77cd-4a3c-8766-984760dc28ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29106e39-77cd-4a3c-8766-984760dc28ad-istiod-ca-cert\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435094    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-socket\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435213 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435158    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-envoy\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435228    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-certs\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435302    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-data\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435337    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-credential-socket\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435385    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-istio-token\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-istio-token\") pod \"29106e39-77cd-4a3c-8766-984760dc28ad\" (UID: \"29106e39-77cd-4a3c-8766-984760dc28ad\") "
Oct 30 20:30:26.435627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435599    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29106e39-77cd-4a3c-8766-984760dc28ad-istiod-ca-cert" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:26.435782 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435749    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-data" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:26.436120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435974    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-certs" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:26.436120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.435994    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-credential-socket" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:26.436120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.436083    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-socket" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:26.436377 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.436142    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.436377 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.436167    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29106e39-77cd-4a3c-8766-984760dc28ad-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.437300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.437268    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29106e39-77cd-4a3c-8766-984760dc28ad-istio-podinfo" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:26.437474 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.437439    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-kube-api-access-r7pld" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "kube-api-access-r7pld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:26.437551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.437526    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-envoy" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:26.437986 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.437973    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-istio-token" pod "29106e39-77cd-4a3c-8766-984760dc28ad" (UID: "29106e39-77cd-4a3c-8766-984760dc28ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:26.536850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.536817    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29106e39-77cd-4a3c-8766-984760dc28ad-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.536850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.536847    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7pld\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-kube-api-access-r7pld\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.536850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.536857    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.537107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.536867    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.537107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.536876    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.537107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.536883    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29106e39-77cd-4a3c-8766-984760dc28ad-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:26.537107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:26.536893    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29106e39-77cd-4a3c-8766-984760dc28ad-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:27.043963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.043928    4519 generic.go:356] "Generic (PLEG): container finished" podID="29106e39-77cd-4a3c-8766-984760dc28ad" containerID="4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e" exitCode=0
Oct 30 20:30:27.044452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.043972    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"
Oct 30 20:30:27.044452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.043999    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" event={"ID":"29106e39-77cd-4a3c-8766-984760dc28ad","Type":"ContainerDied","Data":"4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e"}
Oct 30 20:30:27.044452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.044057    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm" event={"ID":"29106e39-77cd-4a3c-8766-984760dc28ad","Type":"ContainerDied","Data":"22a754b43c68f370cd92fbd1af643504c8bde9ff7e18cb7326635a855a5c7bc1"}
Oct 30 20:30:27.044452 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.044086    4519 scope.go:122] "RemoveContainer" containerID="4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e"
Oct 30 20:30:27.056980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.056958    4519 scope.go:122] "RemoveContainer" containerID="5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0"
Oct 30 20:30:27.067364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.067342    4519 scope.go:122] "RemoveContainer" containerID="4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e"
Oct 30 20:30:27.067663 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:27.067638    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e\": container with ID starting with 4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e not found: ID does not exist" containerID="4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e"
Oct 30 20:30:27.067749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.067672    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e"} err="failed to get container status \"4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e\": rpc error: code = NotFound desc = could not find container \"4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e\": container with ID starting with 4cb5250ea8c63e4a5311ba4f9fce8e10ab45e7a9001691da68fbbfc63271c18e not found: ID does not exist"
Oct 30 20:30:27.067749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.067691    4519 scope.go:122] "RemoveContainer" containerID="5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0"
Oct 30 20:30:27.067934 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:27.067917    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0\": container with ID starting with 5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0 not found: ID does not exist" containerID="5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0"
Oct 30 20:30:27.067996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.067939    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0"} err="failed to get container status \"5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0\": rpc error: code = NotFound desc = could not find container \"5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0\": container with ID starting with 5fa5b92e47498c8eb2ca79886c341bf3b1a8511acd44f2cb5afa8e5aa6c501f0 not found: ID does not exist"
Oct 30 20:30:27.104487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.104448    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"]
Oct 30 20:30:27.116269 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.116243    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-843ea8-openshift-default-6c7f95d5c5-l8qsm"]
Oct 30 20:30:27.611730 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:27.611693    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" path="/var/lib/kubelet/pods/29106e39-77cd-4a3c-8766-984760dc28ad/volumes"
Oct 30 20:30:29.475100 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.474511    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:30:29.570354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570308    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-envoy\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.570578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570382    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-data\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.570578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570429    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f20780-1c13-41d8-a471-9ee518ce0acd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f20780-1c13-41d8-a471-9ee518ce0acd-istiod-ca-cert\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.570578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570501    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-certs\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.570805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570581    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-socket\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.570805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570686    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-podinfo\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.570805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570744    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-credential-socket\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.570805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570786    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-kube-api-access-whqvv\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-kube-api-access-whqvv\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.571042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.570850    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-token\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-token\") pod \"83f20780-1c13-41d8-a471-9ee518ce0acd\" (UID: \"83f20780-1c13-41d8-a471-9ee518ce0acd\") "
Oct 30 20:30:29.571042 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571003    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-certs" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:29.571676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571270    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-data" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:29.571676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571391    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-socket" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:29.571676 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571599    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f20780-1c13-41d8-a471-9ee518ce0acd-istiod-ca-cert" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:29.571910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571842    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.571910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571861    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f20780-1c13-41d8-a471-9ee518ce0acd-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.571910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571875    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.571910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.571889    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.572229 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.572162    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-credential-socket" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:29.574252 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.574225    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-token" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:29.575669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.575619    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-envoy" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:29.575669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.575627    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-kube-api-access-whqvv" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "kube-api-access-whqvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:29.575669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.575628    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-podinfo" pod "83f20780-1c13-41d8-a471-9ee518ce0acd" (UID: "83f20780-1c13-41d8-a471-9ee518ce0acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:29.673272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.673235    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.673272 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.673272    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.673440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.673288    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqvv\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-kube-api-access-whqvv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.673440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.673303    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:29.673440 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:29.673317    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f20780-1c13-41d8-a471-9ee518ce0acd-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:30.075832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.075790    4519 generic.go:356] "Generic (PLEG): container finished" podID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerID="35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c" exitCode=0
Oct 30 20:30:30.076043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.075857    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"
Oct 30 20:30:30.076043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.075875    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" event={"ID":"83f20780-1c13-41d8-a471-9ee518ce0acd","Type":"ContainerDied","Data":"35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c"}
Oct 30 20:30:30.076043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.075937    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv" event={"ID":"83f20780-1c13-41d8-a471-9ee518ce0acd","Type":"ContainerDied","Data":"fbf428cd5f8f7d7dd15e04e0e3fe3e7c685993b83cc46f4df26f774704cce3a7"}
Oct 30 20:30:30.076043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.075962    4519 scope.go:122] "RemoveContainer" containerID="35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c"
Oct 30 20:30:30.088985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.088958    4519 scope.go:122] "RemoveContainer" containerID="35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c"
Oct 30 20:30:30.089390 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:30.089364    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c\": container with ID starting with 35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c not found: ID does not exist" containerID="35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c"
Oct 30 20:30:30.089461 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.089401    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c"} err="failed to get container status \"35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c\": rpc error: code = NotFound desc = could not find container \"35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c\": container with ID starting with 35885d943816ab6039dde3bc7a037a4c059dcbfbfade054aa892c1638a4cde2c not found: ID does not exist"
Oct 30 20:30:30.134939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.134911    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"]
Oct 30 20:30:30.139678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:30.139656    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-8pwrv"]
Oct 30 20:30:31.699608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:31.699551    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" path="/var/lib/kubelet/pods/83f20780-1c13-41d8-a471-9ee518ce0acd/volumes"
Oct 30 20:30:32.087945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:32.087856    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"]
Oct 30 20:30:32.099816 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:32.099777    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-01f7d3-bkl42"]
Oct 30 20:30:32.235411 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:32.235370    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:30:32.716602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:32.716564    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"]
Oct 30 20:30:32.728195 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:32.728165    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-77ac5a-q8dbx"]
Oct 30 20:30:32.934650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:32.934436    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl" podUID="44a97e1f-f884-4434-8104-2acb497c70f7" containerName="main" containerID="cri-o://383f656bb9c2121a0bdbe4c25e1f8bc1aff32dc0e70cb654fe5f2aff5d3e1852" gracePeriod=45
Oct 30 20:30:33.074667 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.074621    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl" podUID="44a97e1f-f884-4434-8104-2acb497c70f7" containerName="main" probeResult="failure" output="Get \"https://10.133.1.59:8000/ready\": dial tcp 10.133.1.59:8000: connect: connection refused"
Oct 30 20:30:33.111053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.111016    4519 generic.go:356] "Generic (PLEG): container finished" podID="44a97e1f-f884-4434-8104-2acb497c70f7" containerID="383f656bb9c2121a0bdbe4c25e1f8bc1aff32dc0e70cb654fe5f2aff5d3e1852" exitCode=0
Oct 30 20:30:33.111201 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.111164    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl" event={"ID":"44a97e1f-f884-4434-8104-2acb497c70f7","Type":"ContainerDied","Data":"383f656bb9c2121a0bdbe4c25e1f8bc1aff32dc0e70cb654fe5f2aff5d3e1852"}
Oct 30 20:30:33.233250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.233222    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:30:33.363325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.361607    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a97e1f-f884-4434-8104-2acb497c70f7-kube-api-access-qj5md\" (UniqueName: \"kubernetes.io/projected/44a97e1f-f884-4434-8104-2acb497c70f7-kube-api-access-qj5md\") pod \"44a97e1f-f884-4434-8104-2acb497c70f7\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") "
Oct 30 20:30:33.363325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.361696    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-model-cache\") pod \"44a97e1f-f884-4434-8104-2acb497c70f7\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") "
Oct 30 20:30:33.363325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.361781    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44a97e1f-f884-4434-8104-2acb497c70f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/44a97e1f-f884-4434-8104-2acb497c70f7-tls-certs\") pod \"44a97e1f-f884-4434-8104-2acb497c70f7\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") "
Oct 30 20:30:33.363325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.361866    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-tmp-dir\") pod \"44a97e1f-f884-4434-8104-2acb497c70f7\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") "
Oct 30 20:30:33.363325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.361953    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-dshm\") pod \"44a97e1f-f884-4434-8104-2acb497c70f7\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") "
Oct 30 20:30:33.363325 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.362012    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-home\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-home\") pod \"44a97e1f-f884-4434-8104-2acb497c70f7\" (UID: \"44a97e1f-f884-4434-8104-2acb497c70f7\") "
Oct 30 20:30:33.363823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.363570    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-tmp-dir" pod "44a97e1f-f884-4434-8104-2acb497c70f7" (UID: "44a97e1f-f884-4434-8104-2acb497c70f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:33.363823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.363584    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-home" pod "44a97e1f-f884-4434-8104-2acb497c70f7" (UID: "44a97e1f-f884-4434-8104-2acb497c70f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:33.363823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.363653    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-model-cache" pod "44a97e1f-f884-4434-8104-2acb497c70f7" (UID: "44a97e1f-f884-4434-8104-2acb497c70f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:33.371836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.368994    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a97e1f-f884-4434-8104-2acb497c70f7-kube-api-access-qj5md" pod "44a97e1f-f884-4434-8104-2acb497c70f7" (UID: "44a97e1f-f884-4434-8104-2acb497c70f7"). InnerVolumeSpecName "kube-api-access-qj5md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:33.371836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.369376    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-dshm" pod "44a97e1f-f884-4434-8104-2acb497c70f7" (UID: "44a97e1f-f884-4434-8104-2acb497c70f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:33.371836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.370619    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a97e1f-f884-4434-8104-2acb497c70f7-tls-certs" pod "44a97e1f-f884-4434-8104-2acb497c70f7" (UID: "44a97e1f-f884-4434-8104-2acb497c70f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:33.377778 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.377750    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"]
Oct 30 20:30:33.385149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.383339    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-284bf6-2j9m5"]
Oct 30 20:30:33.463143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.463101    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:33.463143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.463140    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5md\" (UniqueName: \"kubernetes.io/projected/44a97e1f-f884-4434-8104-2acb497c70f7-kube-api-access-qj5md\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:33.463382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.463156    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:33.463382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.463172    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44a97e1f-f884-4434-8104-2acb497c70f7-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:33.463382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.463185    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:33.463382 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.463197    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44a97e1f-f884-4434-8104-2acb497c70f7-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:33.612602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.612566    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afb7643-aa97-4840-9780-690d3a3538fd" path="/var/lib/kubelet/pods/3afb7643-aa97-4840-9780-690d3a3538fd/volumes"
Oct 30 20:30:33.613186 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.613161    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a195202f-ee30-4ce6-978b-150f09ed08c5" path="/var/lib/kubelet/pods/a195202f-ee30-4ce6-978b-150f09ed08c5/volumes"
Oct 30 20:30:33.613706 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:33.613648    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb2bf3b-af66-4bce-98c3-77906d3d8bf4" path="/var/lib/kubelet/pods/abb2bf3b-af66-4bce-98c3-77906d3d8bf4/volumes"
Oct 30 20:30:34.128005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.127965    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"
Oct 30 20:30:34.128539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.128001    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl" event={"ID":"44a97e1f-f884-4434-8104-2acb497c70f7","Type":"ContainerDied","Data":"c416b3e514aafa67e6b0476791a8f3d4bec780bbda83d3ed25b6342638f8231f"}
Oct 30 20:30:34.128539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.128064    4519 scope.go:122] "RemoveContainer" containerID="383f656bb9c2121a0bdbe4c25e1f8bc1aff32dc0e70cb654fe5f2aff5d3e1852"
Oct 30 20:30:34.188839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.188792    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr" podUID="63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" containerName="main" containerID="cri-o://fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3" gracePeriod=45
Oct 30 20:30:34.198257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.198219    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"]
Oct 30 20:30:34.204734 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.204691    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr" podUID="63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" containerName="main" probeResult="failure" output="Get \"https://10.133.1.60:8000/ready\": dial tcp 10.133.1.60:8000: connect: connection refused"
Oct 30 20:30:34.208079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.208045    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/rate-test-model-5c875d-kserve-647897cc77-87fpl"]
Oct 30 20:30:34.476017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.475986    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:30:34.579000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.578961    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-home\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-home\") pod \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") "
Oct 30 20:30:34.579182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579054    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-dshm\") pod \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") "
Oct 30 20:30:34.579182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579120    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-kube-api-access-p4bxs\" (UniqueName: \"kubernetes.io/projected/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-kube-api-access-p4bxs\") pod \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") "
Oct 30 20:30:34.579182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579151    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tls-certs\") pod \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") "
Oct 30 20:30:34.579308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579217    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-model-cache\") pod \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") "
Oct 30 20:30:34.579308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579253    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tmp-dir\") pod \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\" (UID: \"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0\") "
Oct 30 20:30:34.579308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579296    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-home" pod "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" (UID: "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:34.579763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579734    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:34.579763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579739    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-model-cache" pod "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" (UID: "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:34.579971 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.579933    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tmp-dir" pod "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" (UID: "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:34.581795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.581772    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-dshm" pod "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" (UID: "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:34.581991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.581971    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-kube-api-access-p4bxs" pod "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" (UID: "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0"). InnerVolumeSpecName "kube-api-access-p4bxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:34.582071 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.582057    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tls-certs" pod "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" (UID: "63dd0b1b-6da2-4999-8b21-2f4dd73a15c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:34.680653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.680608    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:34.680653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.680641    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bxs\" (UniqueName: \"kubernetes.io/projected/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-kube-api-access-p4bxs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:34.680653 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.680657    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:34.680984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.680672    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:34.680984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:34.680699    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:35.154628 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.154589    4519 generic.go:356] "Generic (PLEG): container finished" podID="63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" containerID="fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3" exitCode=0
Oct 30 20:30:35.155152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.154647    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"
Oct 30 20:30:35.155152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.154658    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr" event={"ID":"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0","Type":"ContainerDied","Data":"fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3"}
Oct 30 20:30:35.155152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.154710    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr" event={"ID":"63dd0b1b-6da2-4999-8b21-2f4dd73a15c0","Type":"ContainerDied","Data":"b540c54f1570136d9771de43339e1de33c2619326c1b11f9fad3c4e6b8ce0bcb"}
Oct 30 20:30:35.155152 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.154738    4519 scope.go:122] "RemoveContainer" containerID="fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3"
Oct 30 20:30:35.169315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.169293    4519 scope.go:122] "RemoveContainer" containerID="fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3"
Oct 30 20:30:35.169657 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:35.169633    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3\": container with ID starting with fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3 not found: ID does not exist" containerID="fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3"
Oct 30 20:30:35.169741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.169667    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3"} err="failed to get container status \"fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3\": rpc error: code = NotFound desc = could not find container \"fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3\": container with ID starting with fa235ecbc18a8cfde5b9f7fc76bf1b31ea590c5cb59cd2db00e14439742c17d3 not found: ID does not exist"
Oct 30 20:30:35.256618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.256583    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"]
Oct 30 20:30:35.259689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.259664    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d2720/rate-test-model-9d2720-kserve-86b49859cc-sd6hr"]
Oct 30 20:30:35.619114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.619078    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a97e1f-f884-4434-8104-2acb497c70f7" path="/var/lib/kubelet/pods/44a97e1f-f884-4434-8104-2acb497c70f7/volumes"
Oct 30 20:30:35.619626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:35.619604    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" path="/var/lib/kubelet/pods/63dd0b1b-6da2-4999-8b21-2f4dd73a15c0/volumes"
Oct 30 20:30:39.306312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:39.306272    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"]
Oct 30 20:30:39.317560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:39.317526    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5314c2-ht5t9"]
Oct 30 20:30:39.613109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:39.613019    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6107f1-8727-4dc0-b18c-52c2194883ce" path="/var/lib/kubelet/pods/fe6107f1-8727-4dc0-b18c-52c2194883ce/volumes"
Oct 30 20:30:44.203558 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:44.203517    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"]
Oct 30 20:30:44.204422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:44.204365    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy" containerID="cri-o://78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd" gracePeriod=30
Oct 30 20:30:45.207836 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.207787    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"]
Oct 30 20:30:45.209607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.208674    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy" containerID="cri-o://9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666" gracePeriod=30
Oct 30 20:30:45.363628 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.363580    4519 scope.go:122] "RemoveContainer" containerID="5ba5a954db08cc383797f1ae9d54b5d447dd8f1269a6bb25ef7c94d4f277a8a1"
Oct 30 20:30:45.392553 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.392519    4519 scope.go:122] "RemoveContainer" containerID="da11bc93a4b6632d5f3afd73394aebf31a347f4476cc2f6e62bd2881c839d65f"
Oct 30 20:30:45.404682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.404634    4519 scope.go:122] "RemoveContainer" containerID="1d8190abe702efdd744b887ece4b86892cdd169c46601e9020746f4c97a34efa"
Oct 30 20:30:45.788586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.788479    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"]
Oct 30 20:30:45.789181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789152    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c6854c09-8a7c-4668-844d-7ce7ac766060" containerName="main"
Oct 30 20:30:45.789181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789178    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6854c09-8a7c-4668-844d-7ce7ac766060" containerName="main"
Oct 30 20:30:45.789181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789196    4519 cpu_manager.go:414] "containerMap: removing container" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy"
Oct 30 20:30:45.789181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789204    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy"
Oct 30 20:30:45.789181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789213    4519 cpu_manager.go:414] "containerMap: removing container" podUID="29fec69b-da51-432c-accd-cd3a4820b487" containerName="revoke-keys"
Oct 30 20:30:45.789181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789221    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fec69b-da51-432c-accd-cd3a4820b487" containerName="revoke-keys"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789236    4519 cpu_manager.go:414] "containerMap: removing container" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789243    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789250    4519 cpu_manager.go:414] "containerMap: removing container" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789255    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789273    4519 cpu_manager.go:414] "containerMap: removing container" podUID="efe0516c-11f3-4c15-bc16-a60322bdbcbe" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789280    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe0516c-11f3-4c15-bc16-a60322bdbcbe" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789296    4519 cpu_manager.go:414] "containerMap: removing container" podUID="44a97e1f-f884-4434-8104-2acb497c70f7" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789302    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a97e1f-f884-4434-8104-2acb497c70f7" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789308    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a195202f-ee30-4ce6-978b-150f09ed08c5" containerName="revoke-keys"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789312    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a195202f-ee30-4ce6-978b-150f09ed08c5" containerName="revoke-keys"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789321    4519 cpu_manager.go:414] "containerMap: removing container" podUID="63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789325    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789403    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789413    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="29106e39-77cd-4a3c-8766-984760dc28ad" containerName="istio-proxy"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789421    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe0516c-11f3-4c15-bc16-a60322bdbcbe" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789429    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a97e1f-f884-4434-8104-2acb497c70f7" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789438    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a195202f-ee30-4ce6-978b-150f09ed08c5" containerName="revoke-keys"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789447    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="63dd0b1b-6da2-4999-8b21-2f4dd73a15c0" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789456    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6854c09-8a7c-4668-844d-7ce7ac766060" containerName="main"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789477    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f20780-1c13-41d8-a471-9ee518ce0acd" containerName="istio-proxy"
Oct 30 20:30:45.789742 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.789485    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fec69b-da51-432c-accd-cd3a4820b487" containerName="revoke-keys"
Oct 30 20:30:45.795451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.795428    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.810016 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.809980    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:45.810202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.810165    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:30:45.810850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.810823    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"]
Oct 30 20:30:45.822889 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.822858    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmdm\" (UniqueName: \"kubernetes.io/projected/ebce0846-0e12-4009-96eb-8542174ab345-kube-api-access-rkmdm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.823028 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.823006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.823113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.823093    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.924574 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.924536    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmdm\" (UniqueName: \"kubernetes.io/projected/ebce0846-0e12-4009-96eb-8542174ab345-kube-api-access-rkmdm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.924766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.924609    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.924766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.924647    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.925344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.925318    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.927171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.927137    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:45.949402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:45.949370    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmdm\" (UniqueName: \"kubernetes.io/projected/ebce0846-0e12-4009-96eb-8542174ab345-kube-api-access-rkmdm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:46.114547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:46.114460    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:46.434364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:46.434331    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"]
Oct 30 20:30:46.435194 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:46.435163    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebce0846_0e12_4009_96eb_8542174ab345.slice/crio-81bf2b642be6165b46d0069dff403cfba87c12dffeee34ee7f516e5e1b0c9a21 WatchSource:0}: Error finding container 81bf2b642be6165b46d0069dff403cfba87c12dffeee34ee7f516e5e1b0c9a21: Status 404 returned error can't find the container with id 81bf2b642be6165b46d0069dff403cfba87c12dffeee34ee7f516e5e1b0c9a21
Oct 30 20:30:47.284601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:47.284562    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg" event={"ID":"ebce0846-0e12-4009-96eb-8542174ab345","Type":"ContainerStarted","Data":"87c827180231d02eedefdb8531a482e8e16c498339ad8d3504956ee3373afec3"}
Oct 30 20:30:47.284779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:47.284632    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg" event={"ID":"ebce0846-0e12-4009-96eb-8542174ab345","Type":"ContainerStarted","Data":"81bf2b642be6165b46d0069dff403cfba87c12dffeee34ee7f516e5e1b0c9a21"}
Oct 30 20:30:47.327322 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:47.327265    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg" podStartSLOduration=2.327247173 podStartE2EDuration="2.327247173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:47.325525642 +0000 UTC m=+3371.826229367" watchObservedRunningTime="2026-10-30 20:30:47.327247173 +0000 UTC m=+3371.827951300"
Oct 30 20:30:47.899636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:47.899592    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"]
Oct 30 20:30:47.900040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:47.899870    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" containerID="cri-o://99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477" gracePeriod=30
Oct 30 20:30:48.294500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.294433    4519 generic.go:356] "Generic (PLEG): container finished" podID="ebce0846-0e12-4009-96eb-8542174ab345" containerID="87c827180231d02eedefdb8531a482e8e16c498339ad8d3504956ee3373afec3" exitCode=0
Oct 30 20:30:48.294690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.294511    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg" event={"ID":"ebce0846-0e12-4009-96eb-8542174ab345","Type":"ContainerDied","Data":"87c827180231d02eedefdb8531a482e8e16c498339ad8d3504956ee3373afec3"}
Oct 30 20:30:48.426953 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.426915    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"]
Oct 30 20:30:48.431019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.430996    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.452570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.452540    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"]
Oct 30 20:30:48.454838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.454806    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf798\" (UniqueName: \"kubernetes.io/projected/2904cb9f-8c2d-4209-ac0b-dff877013c7b-kube-api-access-xf798\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.455065 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.455044    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.455168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.455125    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.556771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.556689    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.556771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.556767    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.556957 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.556834    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf798\" (UniqueName: \"kubernetes.io/projected/2904cb9f-8c2d-4209-ac0b-dff877013c7b-kube-api-access-xf798\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.557405 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.557370    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.559409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.559382    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.576029 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.576001    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf798\" (UniqueName: \"kubernetes.io/projected/2904cb9f-8c2d-4209-ac0b-dff877013c7b-kube-api-access-xf798\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:48.744718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:48.744635    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:49.031205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.031166    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:30:49.031675 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.031229    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:30:49.062989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.062957    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"]
Oct 30 20:30:49.064271 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:49.064237    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2904cb9f_8c2d_4209_ac0b_dff877013c7b.slice/crio-f2880bec66b9d42869226b4bb260107ac5c1b21c009142655f935f65fc9ba3fa WatchSource:0}: Error finding container f2880bec66b9d42869226b4bb260107ac5c1b21c009142655f935f65fc9ba3fa: Status 404 returned error can't find the container with id f2880bec66b9d42869226b4bb260107ac5c1b21c009142655f935f65fc9ba3fa
Oct 30 20:30:49.302715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.302674    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx" event={"ID":"2904cb9f-8c2d-4209-ac0b-dff877013c7b","Type":"ContainerStarted","Data":"f2880bec66b9d42869226b4bb260107ac5c1b21c009142655f935f65fc9ba3fa"}
Oct 30 20:30:49.461840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.461809    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:49.507392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.503983    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.574842    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-podinfo\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.574916    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebce0846-0e12-4009-96eb-8542174ab345-kube-api-access-rkmdm\" (UniqueName: \"kubernetes.io/projected/ebce0846-0e12-4009-96eb-8542174ab345-kube-api-access-rkmdm\") pod \"ebce0846-0e12-4009-96eb-8542174ab345\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.574942    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-openshift-ca\") pod \"ebce0846-0e12-4009-96eb-8542174ab345\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.574963    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-kube-api-access-jd7tn\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-kube-api-access-jd7tn\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.574984    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-envoy\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.575024    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-xks-ca\") pod \"ebce0846-0e12-4009-96eb-8542174ab345\" (UID: \"ebce0846-0e12-4009-96eb-8542174ab345\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.575049    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb07fbb-698c-459b-a21b-130779cc9afd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb07fbb-698c-459b-a21b-130779cc9afd-istiod-ca-cert\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.575113    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-certs\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.575146    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-data\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.575169    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-socket\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.575197    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-credential-socket\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.575217    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-token\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-token\") pod \"7eb07fbb-698c-459b-a21b-130779cc9afd\" (UID: \"7eb07fbb-698c-459b-a21b-130779cc9afd\") "
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.576335    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-socket" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.576680    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-certs" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.576716    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-data" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.576935    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-credential-socket" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.577010    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb07fbb-698c-459b-a21b-130779cc9afd-istiod-ca-cert" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:49.577549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.577199    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-openshift-ca" pod "ebce0846-0e12-4009-96eb-8542174ab345" (UID: "ebce0846-0e12-4009-96eb-8542174ab345"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:49.580551 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.580504    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-envoy" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:49.580696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.580608    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-podinfo" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:49.580909 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.580871    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebce0846-0e12-4009-96eb-8542174ab345-kube-api-access-rkmdm" pod "ebce0846-0e12-4009-96eb-8542174ab345" (UID: "ebce0846-0e12-4009-96eb-8542174ab345"). InnerVolumeSpecName "kube-api-access-rkmdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:49.581372 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.581347    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-kube-api-access-jd7tn" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "kube-api-access-jd7tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:49.583827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.583780    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-token" pod "7eb07fbb-698c-459b-a21b-130779cc9afd" (UID: "7eb07fbb-698c-459b-a21b-130779cc9afd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:49.584157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.584128    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-xks-ca" pod "ebce0846-0e12-4009-96eb-8542174ab345" (UID: "ebce0846-0e12-4009-96eb-8542174ab345"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:49.677669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677633    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb07fbb-698c-459b-a21b-130779cc9afd-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677671    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677685    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677699    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677711    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677724    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677739    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677753    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmdm\" (UniqueName: \"kubernetes.io/projected/ebce0846-0e12-4009-96eb-8542174ab345-kube-api-access-rkmdm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677767    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677780    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd7tn\" (UniqueName: \"kubernetes.io/projected/7eb07fbb-698c-459b-a21b-130779cc9afd-kube-api-access-jd7tn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677793    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb07fbb-698c-459b-a21b-130779cc9afd-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:49.677913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:49.677808    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0846-0e12-4009-96eb-8542174ab345-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.313033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.312984    4519 generic.go:356] "Generic (PLEG): container finished" podID="2904cb9f-8c2d-4209-ac0b-dff877013c7b" containerID="94bf131ebed7e1c2cb43846c906068489a6b47c00c5e46b14a0840d90f5d9273" exitCode=0
Oct 30 20:30:50.313449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.313078    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx" event={"ID":"2904cb9f-8c2d-4209-ac0b-dff877013c7b","Type":"ContainerDied","Data":"94bf131ebed7e1c2cb43846c906068489a6b47c00c5e46b14a0840d90f5d9273"}
Oct 30 20:30:50.314725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.314703    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"
Oct 30 20:30:50.314888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.314721    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg" event={"ID":"ebce0846-0e12-4009-96eb-8542174ab345","Type":"ContainerDied","Data":"81bf2b642be6165b46d0069dff403cfba87c12dffeee34ee7f516e5e1b0c9a21"}
Oct 30 20:30:50.314888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.314775    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81bf2b642be6165b46d0069dff403cfba87c12dffeee34ee7f516e5e1b0c9a21"
Oct 30 20:30:50.316344 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.316321    4519 generic.go:356] "Generic (PLEG): container finished" podID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerID="78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd" exitCode=0
Oct 30 20:30:50.316456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.316362    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"
Oct 30 20:30:50.316456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.316401    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" event={"ID":"7eb07fbb-698c-459b-a21b-130779cc9afd","Type":"ContainerDied","Data":"78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd"}
Oct 30 20:30:50.316456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.316443    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj" event={"ID":"7eb07fbb-698c-459b-a21b-130779cc9afd","Type":"ContainerDied","Data":"80a07bd11541ff11c459a89b7e913e70bee667d12891e7e4251083f5d9375e48"}
Oct 30 20:30:50.316636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.316502    4519 scope.go:122] "RemoveContainer" containerID="78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd"
Oct 30 20:30:50.327554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.327536    4519 scope.go:122] "RemoveContainer" containerID="78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd"
Oct 30 20:30:50.327877 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:50.327857    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd\": container with ID starting with 78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd not found: ID does not exist" containerID="78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd"
Oct 30 20:30:50.327951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.327894    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd"} err="failed to get container status \"78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd\": rpc error: code = NotFound desc = could not find container \"78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd\": container with ID starting with 78db45649a1abb971883f7c2281d527c34b5f36ea4246fb663950b260e762afd not found: ID does not exist"
Oct 30 20:30:50.345067 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.344987    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"]
Oct 30 20:30:50.354849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.354811    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5c2e32-5kqzq"]
Oct 30 20:30:50.431176 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.429574    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"]
Oct 30 20:30:50.433018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.432988    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-vl9vj"]
Oct 30 20:30:50.485834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.485773    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:50.587703 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.587660    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-data\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.587894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.587743    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-kube-api-access-k2kkp\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.587894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.587814    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-certs\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.588030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.587893    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-envoy\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.588030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.587938    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-credential-socket\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.588030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.587950    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-data" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:50.588030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.587987    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360416d1-6d52-4c39-9f6a-158df599e5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360416d1-6d52-4c39-9f6a-158df599e5a1-istiod-ca-cert\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.588030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588013    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-socket\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.588311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588056    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-podinfo\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.588311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588101    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-token\") pod \"360416d1-6d52-4c39-9f6a-158df599e5a1\" (UID: \"360416d1-6d52-4c39-9f6a-158df599e5a1\") "
Oct 30 20:30:50.588311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588171    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-certs" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:50.588311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588258    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-credential-socket" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:50.588545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588317    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-socket" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:50.588601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588570    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360416d1-6d52-4c39-9f6a-158df599e5a1-istiod-ca-cert" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:50.588601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588582    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.588716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588605    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.588716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588620    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.588716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.588635    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.590215 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.590191    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-envoy" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:50.590448 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.590420    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-token" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:50.590577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.590544    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-podinfo" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:50.590936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.590920    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-kube-api-access-k2kkp" pod "360416d1-6d52-4c39-9f6a-158df599e5a1" (UID: "360416d1-6d52-4c39-9f6a-158df599e5a1"). InnerVolumeSpecName "kube-api-access-k2kkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:50.689554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.689520    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.689554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.689554    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360416d1-6d52-4c39-9f6a-158df599e5a1-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.689554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.689564    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.689782 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.689573    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:50.689782 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:50.689582    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/360416d1-6d52-4c39-9f6a-158df599e5a1-kube-api-access-k2kkp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:51.298449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.298409    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:30:51.298644 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.298494    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:30:51.334299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.334260    4519 generic.go:356] "Generic (PLEG): container finished" podID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerID="9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666" exitCode=0
Oct 30 20:30:51.334852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.334326    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"
Oct 30 20:30:51.334852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.334334    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" event={"ID":"360416d1-6d52-4c39-9f6a-158df599e5a1","Type":"ContainerDied","Data":"9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666"}
Oct 30 20:30:51.334852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.334417    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2" event={"ID":"360416d1-6d52-4c39-9f6a-158df599e5a1","Type":"ContainerDied","Data":"4476ab934734364852bcc317a93379ce083be6c4b126f41f95a6a6f569b62a32"}
Oct 30 20:30:51.334852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.334447    4519 scope.go:122] "RemoveContainer" containerID="9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666"
Oct 30 20:30:51.350916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.350864    4519 scope.go:122] "RemoveContainer" containerID="9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666"
Oct 30 20:30:51.351334 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:51.351302    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666\": container with ID starting with 9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666 not found: ID does not exist" containerID="9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666"
Oct 30 20:30:51.351433 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.351343    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666"} err="failed to get container status \"9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666\": rpc error: code = NotFound desc = could not find container \"9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666\": container with ID starting with 9a7243c7a55d7f051ec230bcd1d08817f1a218c29435973e8505659bac486666 not found: ID does not exist"
Oct 30 20:30:51.411711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.411680    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"]
Oct 30 20:30:51.415085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.415049    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-d5mb2"]
Oct 30 20:30:51.499777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.499751    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:51.599190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.599093    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-xks-ca\") pod \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") "
Oct 30 20:30:51.599354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.599233    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-openshift-ca\") pod \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") "
Oct 30 20:30:51.599354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.599279    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2904cb9f-8c2d-4209-ac0b-dff877013c7b-kube-api-access-xf798\" (UniqueName: \"kubernetes.io/projected/2904cb9f-8c2d-4209-ac0b-dff877013c7b-kube-api-access-xf798\") pod \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\" (UID: \"2904cb9f-8c2d-4209-ac0b-dff877013c7b\") "
Oct 30 20:30:51.599640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.599601    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-openshift-ca" pod "2904cb9f-8c2d-4209-ac0b-dff877013c7b" (UID: "2904cb9f-8c2d-4209-ac0b-dff877013c7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:51.601650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.601618    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2904cb9f-8c2d-4209-ac0b-dff877013c7b-kube-api-access-xf798" pod "2904cb9f-8c2d-4209-ac0b-dff877013c7b" (UID: "2904cb9f-8c2d-4209-ac0b-dff877013c7b"). InnerVolumeSpecName "kube-api-access-xf798". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:51.601776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.601695    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-xks-ca" pod "2904cb9f-8c2d-4209-ac0b-dff877013c7b" (UID: "2904cb9f-8c2d-4209-ac0b-dff877013c7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:51.613064 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.613032    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fec69b-da51-432c-accd-cd3a4820b487" path="/var/lib/kubelet/pods/29fec69b-da51-432c-accd-cd3a4820b487/volumes"
Oct 30 20:30:51.613593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.613576    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" path="/var/lib/kubelet/pods/360416d1-6d52-4c39-9f6a-158df599e5a1/volumes"
Oct 30 20:30:51.614019 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.614005    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" path="/var/lib/kubelet/pods/7eb07fbb-698c-459b-a21b-130779cc9afd/volumes"
Oct 30 20:30:51.700450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.700398    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:51.700450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.700442    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf798\" (UniqueName: \"kubernetes.io/projected/2904cb9f-8c2d-4209-ac0b-dff877013c7b-kube-api-access-xf798\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:51.700450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:51.700458    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2904cb9f-8c2d-4209-ac0b-dff877013c7b-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:52.345790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:52.345760    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"
Oct 30 20:30:52.345790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:52.345772    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx" event={"ID":"2904cb9f-8c2d-4209-ac0b-dff877013c7b","Type":"ContainerDied","Data":"f2880bec66b9d42869226b4bb260107ac5c1b21c009142655f935f65fc9ba3fa"}
Oct 30 20:30:52.346348 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:52.345826    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2880bec66b9d42869226b4bb260107ac5c1b21c009142655f935f65fc9ba3fa"
Oct 30 20:30:53.179892 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.179821    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:30:53.317575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317542    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-kube-api-access-npvn4\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-kube-api-access-npvn4\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.317771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317613    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-socket\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.317771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317659    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istiod-ca-cert\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.317771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317745    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-envoy\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.317942 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317816    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-credential-socket\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.317942 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317885    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-data\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.317942 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317920    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-podinfo\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.318113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317952    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-token\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-token\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.318113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317955    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-socket" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:53.318113 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.317986    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-certs\") pod \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\" (UID: \"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3\") "
Oct 30 20:30:53.318295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.318199    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-certs" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:53.318295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.318215    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istiod-ca-cert" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:53.318295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.318219    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-credential-socket" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:53.318447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.318418    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-data" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:53.319158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.319131    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.319310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.319161    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.319310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.319178    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.319310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.319195    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.319310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.319211    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.320235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.320205    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-envoy" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:53.320501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.320449    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-kube-api-access-npvn4" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "kube-api-access-npvn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:53.320562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.320512    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-podinfo" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:53.321616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.321591    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-token" pod "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" (UID: "d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:53.357810 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.357778    4519 generic.go:356] "Generic (PLEG): container finished" podID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerID="99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477" exitCode=0
Oct 30 20:30:53.358268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.357831    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"
Oct 30 20:30:53.358268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.357860    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" event={"ID":"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3","Type":"ContainerDied","Data":"99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477"}
Oct 30 20:30:53.358268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.357936    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm" event={"ID":"d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3","Type":"ContainerDied","Data":"1ef627c33e49822d7255f122a3b65db3b94e08ded6f1bf9780ddba27f22046b7"}
Oct 30 20:30:53.358268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.357973    4519 scope.go:122] "RemoveContainer" containerID="99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477"
Oct 30 20:30:53.370043 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.370022    4519 scope.go:122] "RemoveContainer" containerID="604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef"
Oct 30 20:30:53.380946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.380925    4519 scope.go:122] "RemoveContainer" containerID="99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477"
Oct 30 20:30:53.381235 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:53.381215    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477\": container with ID starting with 99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477 not found: ID does not exist" containerID="99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477"
Oct 30 20:30:53.381282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.381244    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477"} err="failed to get container status \"99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477\": rpc error: code = NotFound desc = could not find container \"99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477\": container with ID starting with 99f5306779955b3b7afe1918c72e3bc834eee11ea3d56a934bbf70f1250bb477 not found: ID does not exist"
Oct 30 20:30:53.381282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.381261    4519 scope.go:122] "RemoveContainer" containerID="604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef"
Oct 30 20:30:53.381537 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:30:53.381517    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef\": container with ID starting with 604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef not found: ID does not exist" containerID="604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef"
Oct 30 20:30:53.381607 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.381542    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef"} err="failed to get container status \"604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef\": rpc error: code = NotFound desc = could not find container \"604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef\": container with ID starting with 604a470dc7b915daac57cbdfc3a63d5fcc686c1f2150aaf2a3e4aa90bd6734ef not found: ID does not exist"
Oct 30 20:30:53.418595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.418561    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"]
Oct 30 20:30:53.419974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.419955    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.420085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.419977    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.420085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.419988    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.420085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.419997    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvn4\" (UniqueName: \"kubernetes.io/projected/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3-kube-api-access-npvn4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:30:53.422302 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.422278    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5314c2-openshift-default-7d8f88d9cf-4sgmm"]
Oct 30 20:30:53.613781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:53.613697    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" path="/var/lib/kubelet/pods/d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3/volumes"
Oct 30 20:30:58.520771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.520725    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"]
Oct 30 20:30:58.521371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521346    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2904cb9f-8c2d-4209-ac0b-dff877013c7b" containerName="revoke-keys"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521373    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2904cb9f-8c2d-4209-ac0b-dff877013c7b" containerName="revoke-keys"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521409    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521418    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521434    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521443    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521453    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ebce0846-0e12-4009-96eb-8542174ab345" containerName="revoke-keys"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521460    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebce0846-0e12-4009-96eb-8542174ab345" containerName="revoke-keys"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521512    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy"
Oct 30 20:30:58.521519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521520    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521537    4519 cpu_manager.go:414] "containerMap: removing container" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521545    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521663    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb07fbb-698c-459b-a21b-130779cc9afd" containerName="istio-proxy"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521675    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="360416d1-6d52-4c39-9f6a-158df599e5a1" containerName="istio-proxy"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521686    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2904cb9f-8c2d-4209-ac0b-dff877013c7b" containerName="revoke-keys"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521699    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521710    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c9ead9-68b8-41cc-8ef8-6f4cdb63cee3" containerName="istio-proxy"
Oct 30 20:30:58.521989 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.521718    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebce0846-0e12-4009-96eb-8542174ab345" containerName="revoke-keys"
Oct 30 20:30:58.527012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.526988    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.541915 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.541893    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:30:58.554062 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.554032    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:58.558904 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.558873    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"]
Oct 30 20:30:58.695157 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.695109    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxgk2\" (UniqueName: \"kubernetes.io/projected/3dd74364-6a05-4176-9624-e3a91e163f97-kube-api-access-gxgk2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.695362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.695241    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.695362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.695296    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.796578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.796494    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxgk2\" (UniqueName: \"kubernetes.io/projected/3dd74364-6a05-4176-9624-e3a91e163f97-kube-api-access-gxgk2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.796578 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.796564    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.796790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.796609    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.797234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.797211    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.799080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.799057    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.835545 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.835510    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxgk2\" (UniqueName: \"kubernetes.io/projected/3dd74364-6a05-4176-9624-e3a91e163f97-kube-api-access-gxgk2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:58.841126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:58.841098    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:30:59.154205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:59.154172    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"]
Oct 30 20:30:59.155247 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:30:59.155210    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd74364_6a05_4176_9624_e3a91e163f97.slice/crio-e1374704500373a831335754710fec1ff9022bd646e2a33969e9144e1a4d8132 WatchSource:0}: Error finding container e1374704500373a831335754710fec1ff9022bd646e2a33969e9144e1a4d8132: Status 404 returned error can't find the container with id e1374704500373a831335754710fec1ff9022bd646e2a33969e9144e1a4d8132
Oct 30 20:30:59.410254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:30:59.410156    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz" event={"ID":"3dd74364-6a05-4176-9624-e3a91e163f97","Type":"ContainerStarted","Data":"e1374704500373a831335754710fec1ff9022bd646e2a33969e9144e1a4d8132"}
Oct 30 20:31:00.419031 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:00.418997    4519 generic.go:356] "Generic (PLEG): container finished" podID="3dd74364-6a05-4176-9624-e3a91e163f97" containerID="3fe1f390ae757fc4f6bf9e9afdda2eebfcbd25fdd2dc27d9624719e82c7dc485" exitCode=0
Oct 30 20:31:00.419487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:00.419083    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz" event={"ID":"3dd74364-6a05-4176-9624-e3a91e163f97","Type":"ContainerDied","Data":"3fe1f390ae757fc4f6bf9e9afdda2eebfcbd25fdd2dc27d9624719e82c7dc485"}
Oct 30 20:31:00.779515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:00.779363    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"]
Oct 30 20:31:00.784514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:00.784460    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5c875d-l7lbx"]
Oct 30 20:31:01.570610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.570575    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:31:01.611387 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.611345    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2904cb9f-8c2d-4209-ac0b-dff877013c7b" path="/var/lib/kubelet/pods/2904cb9f-8c2d-4209-ac0b-dff877013c7b/volumes"
Oct 30 20:31:01.729585 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.729461    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd74364-6a05-4176-9624-e3a91e163f97-kube-api-access-gxgk2\" (UniqueName: \"kubernetes.io/projected/3dd74364-6a05-4176-9624-e3a91e163f97-kube-api-access-gxgk2\") pod \"3dd74364-6a05-4176-9624-e3a91e163f97\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") "
Oct 30 20:31:01.729767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.729606    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-xks-ca\") pod \"3dd74364-6a05-4176-9624-e3a91e163f97\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") "
Oct 30 20:31:01.729767 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.729658    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-openshift-ca\") pod \"3dd74364-6a05-4176-9624-e3a91e163f97\" (UID: \"3dd74364-6a05-4176-9624-e3a91e163f97\") "
Oct 30 20:31:01.730121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.730090    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-openshift-ca" pod "3dd74364-6a05-4176-9624-e3a91e163f97" (UID: "3dd74364-6a05-4176-9624-e3a91e163f97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:01.731963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.731938    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd74364-6a05-4176-9624-e3a91e163f97-kube-api-access-gxgk2" pod "3dd74364-6a05-4176-9624-e3a91e163f97" (UID: "3dd74364-6a05-4176-9624-e3a91e163f97"). InnerVolumeSpecName "kube-api-access-gxgk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:01.731963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.731942    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-xks-ca" pod "3dd74364-6a05-4176-9624-e3a91e163f97" (UID: "3dd74364-6a05-4176-9624-e3a91e163f97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:01.830933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.830891    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxgk2\" (UniqueName: \"kubernetes.io/projected/3dd74364-6a05-4176-9624-e3a91e163f97-kube-api-access-gxgk2\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:01.830933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.830929    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:01.830933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:01.830939    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd74364-6a05-4176-9624-e3a91e163f97-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:02.220941 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:02.220899    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:31:02.434250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:02.434216    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"
Oct 30 20:31:02.434250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:02.434227    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz" event={"ID":"3dd74364-6a05-4176-9624-e3a91e163f97","Type":"ContainerDied","Data":"e1374704500373a831335754710fec1ff9022bd646e2a33969e9144e1a4d8132"}
Oct 30 20:31:02.434510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:02.434281    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1374704500373a831335754710fec1ff9022bd646e2a33969e9144e1a4d8132"
Oct 30 20:31:03.031514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:03.031458    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"]
Oct 30 20:31:03.035725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:03.035698    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-945865d9-sqf2d"]
Oct 30 20:31:03.043103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:03.043071    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"]
Oct 30 20:31:03.045807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:03.045785    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c05905-9mchk"]
Oct 30 20:31:03.611310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:03.611273    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5764e96d-e897-40d5-a387-c8aec9042b43" path="/var/lib/kubelet/pods/5764e96d-e897-40d5-a387-c8aec9042b43/volumes"
Oct 30 20:31:03.611780 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:03.611764    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d9516f-c965-469c-a75b-346b0684801d" path="/var/lib/kubelet/pods/71d9516f-c965-469c-a75b-346b0684801d/volumes"
Oct 30 20:31:05.192244 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:05.192202    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"]
Oct 30 20:31:05.203814 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:05.203782    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d2720-vzqgg"]
Oct 30 20:31:05.612798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:05.612722    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebce0846-0e12-4009-96eb-8542174ab345" path="/var/lib/kubelet/pods/ebce0846-0e12-4009-96eb-8542174ab345/volumes"
Oct 30 20:31:07.034331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:07.034290    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"]
Oct 30 20:31:07.047851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:07.047815    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3b1543-98b2t"]
Oct 30 20:31:07.611367 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:07.611329    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6468a513-718f-4a96-8f4a-8fc872a03026" path="/var/lib/kubelet/pods/6468a513-718f-4a96-8f4a-8fc872a03026/volumes"
Oct 30 20:31:08.029819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.029782    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"]
Oct 30 20:31:08.041111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.041081    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb56acd5-z7ctg"]
Oct 30 20:31:08.184688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.184649    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"]
Oct 30 20:31:08.185027 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.184958    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy" containerID="cri-o://5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64" gracePeriod=30
Oct 30 20:31:08.923947 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.923910    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"]
Oct 30 20:31:08.924358 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.924345    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3dd74364-6a05-4176-9624-e3a91e163f97" containerName="revoke-keys"
Oct 30 20:31:08.924407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.924360    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd74364-6a05-4176-9624-e3a91e163f97" containerName="revoke-keys"
Oct 30 20:31:08.924447 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.924435    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd74364-6a05-4176-9624-e3a91e163f97" containerName="revoke-keys"
Oct 30 20:31:08.929296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.929271    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:08.944859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.944835    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:08.960311 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.960277    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"]
Oct 30 20:31:08.960592 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:08.960421    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:31:09.006933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.006897    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.007114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.006952    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhnj\" (UniqueName: \"kubernetes.io/projected/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-kube-api-access-zlhnj\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.007114 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.007065    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.108116 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.108074    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.108535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.108143    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhnj\" (UniqueName: \"kubernetes.io/projected/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-kube-api-access-zlhnj\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.108535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.108205    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.108822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.108787    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.110663 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.110643    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.128060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.128027    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhnj\" (UniqueName: \"kubernetes.io/projected/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-kube-api-access-zlhnj\") pod \"maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.243360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.243268    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:09.580929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.580897    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"]
Oct 30 20:31:09.582122 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:31:09.582086    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4caf50_3b35_4b94_a892_f56ca5e9f1c4.slice/crio-76105d015ea85ce67fe616a6d55e1a06bd6f915541ba3fec26db7a765db35229 WatchSource:0}: Error finding container 76105d015ea85ce67fe616a6d55e1a06bd6f915541ba3fec26db7a765db35229: Status 404 returned error can't find the container with id 76105d015ea85ce67fe616a6d55e1a06bd6f915541ba3fec26db7a765db35229
Oct 30 20:31:09.611174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:09.611145    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cba18c-e6d5-428e-a302-d6a90e6e87d5" path="/var/lib/kubelet/pods/f1cba18c-e6d5-428e-a302-d6a90e6e87d5/volumes"
Oct 30 20:31:10.508315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:10.508276    4519 generic.go:356] "Generic (PLEG): container finished" podID="4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" containerID="e612f6b268829fb7d1ecc42e4d38699ed4b6f69ead7ac6178c7a838a807e0517" exitCode=0
Oct 30 20:31:10.508770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:10.508361    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt" event={"ID":"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4","Type":"ContainerDied","Data":"e612f6b268829fb7d1ecc42e4d38699ed4b6f69ead7ac6178c7a838a807e0517"}
Oct 30 20:31:10.508770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:10.508430    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt" event={"ID":"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4","Type":"ContainerStarted","Data":"76105d015ea85ce67fe616a6d55e1a06bd6f915541ba3fec26db7a765db35229"}
Oct 30 20:31:11.662636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.662605    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:11.737936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.737895    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-openshift-ca\") pod \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") "
Oct 30 20:31:11.738151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.738017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-xks-ca\") pod \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") "
Oct 30 20:31:11.738151 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.738105    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-kube-api-access-zlhnj\" (UniqueName: \"kubernetes.io/projected/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-kube-api-access-zlhnj\") pod \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\" (UID: \"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4\") "
Oct 30 20:31:11.738331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.738307    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-openshift-ca" pod "4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" (UID: "4b4caf50-3b35-4b94-a892-f56ca5e9f1c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:11.738532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.738510    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:11.740634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.740598    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-kube-api-access-zlhnj" pod "4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" (UID: "4b4caf50-3b35-4b94-a892-f56ca5e9f1c4"). InnerVolumeSpecName "kube-api-access-zlhnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:11.740765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.740596    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-xks-ca" pod "4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" (UID: "4b4caf50-3b35-4b94-a892-f56ca5e9f1c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:11.839911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.839821    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:11.839911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:11.839853    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhnj\" (UniqueName: \"kubernetes.io/projected/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4-kube-api-access-zlhnj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:12.540491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:12.539900    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt" event={"ID":"4b4caf50-3b35-4b94-a892-f56ca5e9f1c4","Type":"ContainerDied","Data":"76105d015ea85ce67fe616a6d55e1a06bd6f915541ba3fec26db7a765db35229"}
Oct 30 20:31:12.540491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:12.539968    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76105d015ea85ce67fe616a6d55e1a06bd6f915541ba3fec26db7a765db35229"
Oct 30 20:31:12.540491 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:12.540038    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"
Oct 30 20:31:12.604520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:12.602666    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"]
Oct 30 20:31:12.604520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:12.602922    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" containerID="cri-o://9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c" gracePeriod=30
Oct 30 20:31:13.043003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.042960    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"]
Oct 30 20:31:13.048095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.048063    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-27a4d3-shrgz"]
Oct 30 20:31:13.341637 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.341601    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"]
Oct 30 20:31:13.342072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.342058    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" containerName="revoke-keys"
Oct 30 20:31:13.342121 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.342073    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" containerName="revoke-keys"
Oct 30 20:31:13.342155 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.342133    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" containerName="revoke-keys"
Oct 30 20:31:13.345526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.345502    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.366268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.363909    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"]
Oct 30 20:31:13.390945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.390892    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:13.391094 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.390982    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:31:13.465935 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.465901    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj75q\" (UniqueName: \"kubernetes.io/projected/8301f28f-d413-45c6-9a56-3478f794c2ba-kube-api-access-bj75q\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.466093 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.465962    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.466190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.466161    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.481059 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.481031    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:31:13.550526 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.550485    4519 generic.go:356] "Generic (PLEG): container finished" podID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerID="5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64" exitCode=0
Oct 30 20:31:13.551040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.551010    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" event={"ID":"1781aa1d-20cb-44a8-b7f8-0004676efc46","Type":"ContainerDied","Data":"5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64"}
Oct 30 20:31:13.551228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.551210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s" event={"ID":"1781aa1d-20cb-44a8-b7f8-0004676efc46","Type":"ContainerDied","Data":"9cce90c4e940e914f804a0c0254c884ade8610bdde5e38766241a1fcc73cc88a"}
Oct 30 20:31:13.551353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.551339    4519 scope.go:122] "RemoveContainer" containerID="5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64"
Oct 30 20:31:13.551655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.551633    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"
Oct 30 20:31:13.565222 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.565199    4519 scope.go:122] "RemoveContainer" containerID="3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46"
Oct 30 20:31:13.566984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.566957    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-token\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-token\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.567225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.567203    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-certs\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.567385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.567365    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1781aa1d-20cb-44a8-b7f8-0004676efc46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1781aa1d-20cb-44a8-b7f8-0004676efc46-istiod-ca-cert\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.567625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.567603    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-socket\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.567814 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.567796    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-data\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.567982 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.567952    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-envoy\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.568117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.568102    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-credential-socket\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.568282 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.568267    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-kube-api-access-fqzwp\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-kube-api-access-fqzwp\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.568437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.568422    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-podinfo\") pod \"1781aa1d-20cb-44a8-b7f8-0004676efc46\" (UID: \"1781aa1d-20cb-44a8-b7f8-0004676efc46\") "
Oct 30 20:31:13.568827 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.568802    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj75q\" (UniqueName: \"kubernetes.io/projected/8301f28f-d413-45c6-9a56-3478f794c2ba-kube-api-access-bj75q\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.569619 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.569597    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.570360 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.570337    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.571339 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.571178    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.571504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.567713    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-certs" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:13.571613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.567876    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1781aa1d-20cb-44a8-b7f8-0004676efc46-istiod-ca-cert" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:13.571719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.568127    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-socket" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:13.571822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.568320    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-data" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:13.571822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.569486    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-credential-socket" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:13.574254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.574037    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-kube-api-access-fqzwp" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "kube-api-access-fqzwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:13.574254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.574050    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-token" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:13.574254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.574152    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-envoy" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:13.574254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.574197    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.574727 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.574695    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-podinfo" pod "1781aa1d-20cb-44a8-b7f8-0004676efc46" (UID: "1781aa1d-20cb-44a8-b7f8-0004676efc46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:13.588857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.588821    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj75q\" (UniqueName: \"kubernetes.io/projected/8301f28f-d413-45c6-9a56-3478f794c2ba-kube-api-access-bj75q\") pod \"maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.598636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.598604    4519 scope.go:122] "RemoveContainer" containerID="5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64"
Oct 30 20:31:13.599036 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:13.599009    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64\": container with ID starting with 5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64 not found: ID does not exist" containerID="5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64"
Oct 30 20:31:13.599148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.599046    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64"} err="failed to get container status \"5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64\": rpc error: code = NotFound desc = could not find container \"5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64\": container with ID starting with 5f9c352470071105461a78e48e3239becee101542eda3e95d890be4e7668bf64 not found: ID does not exist"
Oct 30 20:31:13.599148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.599067    4519 scope.go:122] "RemoveContainer" containerID="3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46"
Oct 30 20:31:13.599416 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:13.599393    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46\": container with ID starting with 3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46 not found: ID does not exist" containerID="3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46"
Oct 30 20:31:13.599497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.599423    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46"} err="failed to get container status \"3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46\": rpc error: code = NotFound desc = could not find container \"3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46\": container with ID starting with 3e5af10b22896030b1b4ad1aec5e72986ea1de77ce69bdd55fd74e4d41b0bd46 not found: ID does not exist"
Oct 30 20:31:13.612539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.612381    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd74364-6a05-4176-9624-e3a91e163f97" path="/var/lib/kubelet/pods/3dd74364-6a05-4176-9624-e3a91e163f97/volumes"
Oct 30 20:31:13.652529 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.652491    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"]
Oct 30 20:31:13.662530 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.662498    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:13.672362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672327    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672362 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672365    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672380    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672394    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672406    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqzwp\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-kube-api-access-fqzwp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672421    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672434    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1781aa1d-20cb-44a8-b7f8-0004676efc46-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672445    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1781aa1d-20cb-44a8-b7f8-0004676efc46-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.672626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.672457    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1781aa1d-20cb-44a8-b7f8-0004676efc46-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:13.896172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.896088    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"]
Oct 30 20:31:13.898895 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.898868    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-f6g9s"]
Oct 30 20:31:13.984449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:13.984415    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"]
Oct 30 20:31:13.985954 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:31:13.985922    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8301f28f_d413_45c6_9a56_3478f794c2ba.slice/crio-94653784c21b8d0ebb57bdf6e35b59c4cc1988b491c16e45a3eac4ca0c3c4d40 WatchSource:0}: Error finding container 94653784c21b8d0ebb57bdf6e35b59c4cc1988b491c16e45a3eac4ca0c3c4d40: Status 404 returned error can't find the container with id 94653784c21b8d0ebb57bdf6e35b59c4cc1988b491c16e45a3eac4ca0c3c4d40
Oct 30 20:31:14.568695 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:14.568593    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z" event={"ID":"8301f28f-d413-45c6-9a56-3478f794c2ba","Type":"ContainerStarted","Data":"a30af797c1f780a9ba34db8dec79c132354ced33bc3c28834ae90e8c8cddeecd"}
Oct 30 20:31:14.569314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:14.569283    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z" event={"ID":"8301f28f-d413-45c6-9a56-3478f794c2ba","Type":"ContainerStarted","Data":"94653784c21b8d0ebb57bdf6e35b59c4cc1988b491c16e45a3eac4ca0c3c4d40"}
Oct 30 20:31:14.611185 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:14.611124    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z" podStartSLOduration=1.611107612 podStartE2EDuration="1.611107612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:14.608603919 +0000 UTC m=+3399.109307644" watchObservedRunningTime="2026-10-30 20:31:14.611107612 +0000 UTC m=+3399.111811337"
Oct 30 20:31:15.057061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:15.057024    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:31:15.057254 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:15.057081    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:31:15.580640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:15.580603    4519 generic.go:356] "Generic (PLEG): container finished" podID="8301f28f-d413-45c6-9a56-3478f794c2ba" containerID="a30af797c1f780a9ba34db8dec79c132354ced33bc3c28834ae90e8c8cddeecd" exitCode=0
Oct 30 20:31:15.581118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:15.580668    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z" event={"ID":"8301f28f-d413-45c6-9a56-3478f794c2ba","Type":"ContainerDied","Data":"a30af797c1f780a9ba34db8dec79c132354ced33bc3c28834ae90e8c8cddeecd"}
Oct 30 20:31:15.612133 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:15.612100    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" path="/var/lib/kubelet/pods/1781aa1d-20cb-44a8-b7f8-0004676efc46/volumes"
Oct 30 20:31:16.732298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.732270    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:16.806402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.806364    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-xks-ca\") pod \"8301f28f-d413-45c6-9a56-3478f794c2ba\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") "
Oct 30 20:31:16.806643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.806459    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8301f28f-d413-45c6-9a56-3478f794c2ba-kube-api-access-bj75q\" (UniqueName: \"kubernetes.io/projected/8301f28f-d413-45c6-9a56-3478f794c2ba-kube-api-access-bj75q\") pod \"8301f28f-d413-45c6-9a56-3478f794c2ba\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") "
Oct 30 20:31:16.806643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.806576    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-openshift-ca\") pod \"8301f28f-d413-45c6-9a56-3478f794c2ba\" (UID: \"8301f28f-d413-45c6-9a56-3478f794c2ba\") "
Oct 30 20:31:16.806976 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.806947    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-openshift-ca" pod "8301f28f-d413-45c6-9a56-3478f794c2ba" (UID: "8301f28f-d413-45c6-9a56-3478f794c2ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:16.808740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.808709    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8301f28f-d413-45c6-9a56-3478f794c2ba-kube-api-access-bj75q" pod "8301f28f-d413-45c6-9a56-3478f794c2ba" (UID: "8301f28f-d413-45c6-9a56-3478f794c2ba"). InnerVolumeSpecName "kube-api-access-bj75q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:16.808856 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.808835    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-xks-ca" pod "8301f28f-d413-45c6-9a56-3478f794c2ba" (UID: "8301f28f-d413-45c6-9a56-3478f794c2ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:16.907869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.907780    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj75q\" (UniqueName: \"kubernetes.io/projected/8301f28f-d413-45c6-9a56-3478f794c2ba-kube-api-access-bj75q\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:16.907869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.907812    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:16.907869 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:16.907826    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8301f28f-d413-45c6-9a56-3478f794c2ba-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:17.599738 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:17.599703    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"
Oct 30 20:31:17.599931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:17.599740    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z" event={"ID":"8301f28f-d413-45c6-9a56-3478f794c2ba","Type":"ContainerDied","Data":"94653784c21b8d0ebb57bdf6e35b59c4cc1988b491c16e45a3eac4ca0c3c4d40"}
Oct 30 20:31:17.599931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:17.599795    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94653784c21b8d0ebb57bdf6e35b59c4cc1988b491c16e45a3eac4ca0c3c4d40"
Oct 30 20:31:17.897414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:17.897380    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:31:18.019009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.018971    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-data\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019048    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-kube-api-access-r78d7\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-kube-api-access-r78d7\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019074    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-podinfo\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019097    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-credential-socket\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019136    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-envoy\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019202    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-socket\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019259    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-certs\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019333    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a08deb-6080-41c4-baea-37afb9d69f36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a08deb-6080-41c4-baea-37afb9d69f36-istiod-ca-cert\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019334    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-data" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:18.019450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019395    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-token\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-token\") pod \"c6a08deb-6080-41c4-baea-37afb9d69f36\" (UID: \"c6a08deb-6080-41c4-baea-37afb9d69f36\") "
Oct 30 20:31:18.019795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019506    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-credential-socket" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:18.020076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019850    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.020076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019874    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.020076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019951    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-certs" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:18.020076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.019964    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-socket" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:18.020076 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.020016    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a08deb-6080-41c4-baea-37afb9d69f36-istiod-ca-cert" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:18.021731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.021702    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-podinfo" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:18.021731 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.021720    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-token" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:18.021945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.021928    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-kube-api-access-r78d7" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "kube-api-access-r78d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:18.022005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.021985    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-envoy" pod "c6a08deb-6080-41c4-baea-37afb9d69f36" (UID: "c6a08deb-6080-41c4-baea-37afb9d69f36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:18.070614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.070581    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"]
Oct 30 20:31:18.120871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.120832    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.120871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.120870    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.121089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.120885    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a08deb-6080-41c4-baea-37afb9d69f36-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.121089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.120899    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.121089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.120912    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78d7\" (UniqueName: \"kubernetes.io/projected/c6a08deb-6080-41c4-baea-37afb9d69f36-kube-api-access-r78d7\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.121089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.120927    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.121089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.120940    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a08deb-6080-41c4-baea-37afb9d69f36-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:18.609331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.609287    4519 generic.go:356] "Generic (PLEG): container finished" podID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerID="9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c" exitCode=0
Oct 30 20:31:18.609528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.609353    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"
Oct 30 20:31:18.609528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.609369    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" event={"ID":"c6a08deb-6080-41c4-baea-37afb9d69f36","Type":"ContainerDied","Data":"9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c"}
Oct 30 20:31:18.609528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.609428    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht" event={"ID":"c6a08deb-6080-41c4-baea-37afb9d69f36","Type":"ContainerDied","Data":"eba5ffeca949906fde34642ca627f81e9b15c218694179398ee801f84015aba2"}
Oct 30 20:31:18.609528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.609459    4519 scope.go:122] "RemoveContainer" containerID="9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c"
Oct 30 20:31:18.621527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.621505    4519 scope.go:122] "RemoveContainer" containerID="b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982"
Oct 30 20:31:18.632348 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.632315    4519 scope.go:122] "RemoveContainer" containerID="9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c"
Oct 30 20:31:18.632756 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:18.632729    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c\": container with ID starting with 9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c not found: ID does not exist" containerID="9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c"
Oct 30 20:31:18.632867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.632770    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c"} err="failed to get container status \"9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c\": rpc error: code = NotFound desc = could not find container \"9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c\": container with ID starting with 9e0a1b1659d5c380ecdec857f95a837f6211139a92725da46ed9743409ab2f7c not found: ID does not exist"
Oct 30 20:31:18.632867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.632798    4519 scope.go:122] "RemoveContainer" containerID="b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982"
Oct 30 20:31:18.633159 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:18.633123    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982\": container with ID starting with b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982 not found: ID does not exist" containerID="b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982"
Oct 30 20:31:18.633273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.633161    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982"} err="failed to get container status \"b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982\": rpc error: code = NotFound desc = could not find container \"b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982\": container with ID starting with b352c1239db206f391bfa8f543e30664b4cb289d9b78aca8ce905081e1b45982 not found: ID does not exist"
Oct 30 20:31:18.662548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.662509    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"]
Oct 30 20:31:18.672741 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:18.672713    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-vhfht"]
Oct 30 20:31:19.611299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:19.611264    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" path="/var/lib/kubelet/pods/c6a08deb-6080-41c4-baea-37afb9d69f36/volumes"
Oct 30 20:31:21.062335 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.062299    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"]
Oct 30 20:31:21.062839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.062630    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" containerID="cri-o://f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31" gracePeriod=30
Oct 30 20:31:21.754172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754132    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"]
Oct 30 20:31:21.754624 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754605    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy"
Oct 30 20:31:21.754686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754626    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy"
Oct 30 20:31:21.754686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754641    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy"
Oct 30 20:31:21.754686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754647    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy"
Oct 30 20:31:21.754686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754659    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy"
Oct 30 20:31:21.754686 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754666    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754697    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754703    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754709    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8301f28f-d413-45c6-9a56-3478f794c2ba" containerName="revoke-keys"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754714    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8301f28f-d413-45c6-9a56-3478f794c2ba" containerName="revoke-keys"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754800    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754810    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8301f28f-d413-45c6-9a56-3478f794c2ba" containerName="revoke-keys"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754819    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754830    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1781aa1d-20cb-44a8-b7f8-0004676efc46" containerName="istio-proxy"
Oct 30 20:31:21.754871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.754839    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a08deb-6080-41c4-baea-37afb9d69f36" containerName="istio-proxy"
Oct 30 20:31:21.760074 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.760053    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.774639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.774611    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"]
Oct 30 20:31:21.779542 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.779512    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"]
Oct 30 20:31:21.779677 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.779637    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.782772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.782738    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:21.782912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.782875    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:31:21.796803 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.796770    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"]
Oct 30 20:31:21.861126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.861088    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.861310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.861142    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4pzz\" (UniqueName: \"kubernetes.io/projected/a8a147ff-5aca-4876-952a-6c05cb21b6c4-kube-api-access-s4pzz\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.861310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.861256    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.861414 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.861303    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgskq\" (UniqueName: \"kubernetes.io/projected/4633c0f2-30f0-48e9-b494-9b7d89323df2-kube-api-access-tgskq\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.861449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.861411    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.861507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.861492    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.962588 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.962545    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.962797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.962609    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.962797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.962635    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4pzz\" (UniqueName: \"kubernetes.io/projected/a8a147ff-5aca-4876-952a-6c05cb21b6c4-kube-api-access-s4pzz\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.962797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.962679    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.962797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.962703    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgskq\" (UniqueName: \"kubernetes.io/projected/4633c0f2-30f0-48e9-b494-9b7d89323df2-kube-api-access-tgskq\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.962797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.962758    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.963435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.963401    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.963575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.963496    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.965171 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.965147    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:21.965264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.965227    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.982250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.982217    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgskq\" (UniqueName: \"kubernetes.io/projected/4633c0f2-30f0-48e9-b494-9b7d89323df2-kube-api-access-tgskq\") pod \"maas-api-revoke-sub-67617f59-9wzph\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:21.982409 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:21.982386    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4pzz\" (UniqueName: \"kubernetes.io/projected/a8a147ff-5aca-4876-952a-6c05cb21b6c4-kube-api-access-s4pzz\") pod \"maas-api-revoke-sub-f09fec3d-m2lcb\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:22.077570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.077486    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:22.102008 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.101980    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:22.387615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.387583    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"]
Oct 30 20:31:22.389430 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:31:22.389403    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4633c0f2_30f0_48e9_b494_9b7d89323df2.slice/crio-d430318a5665df51b99a0f856ad2808976fe9ec9fae2f117c3223e746525296c WatchSource:0}: Error finding container d430318a5665df51b99a0f856ad2808976fe9ec9fae2f117c3223e746525296c: Status 404 returned error can't find the container with id d430318a5665df51b99a0f856ad2808976fe9ec9fae2f117c3223e746525296c
Oct 30 20:31:22.417698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.417667    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"]
Oct 30 20:31:22.436966 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:31:22.436933    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a147ff_5aca_4876_952a_6c05cb21b6c4.slice/crio-54b17ff5145451bb39f03d83533662fb979760c3836172a9aa020b844eda5108 WatchSource:0}: Error finding container 54b17ff5145451bb39f03d83533662fb979760c3836172a9aa020b844eda5108: Status 404 returned error can't find the container with id 54b17ff5145451bb39f03d83533662fb979760c3836172a9aa020b844eda5108
Oct 30 20:31:22.577191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.577157    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:31:22.577315 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.577213    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:31:22.640900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.640807    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph" event={"ID":"4633c0f2-30f0-48e9-b494-9b7d89323df2","Type":"ContainerStarted","Data":"15f0311e047aac9a2350e13a4ed50a8812ae1d918f07ec05e22799a89518a82e"}
Oct 30 20:31:22.640900 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.640868    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph" event={"ID":"4633c0f2-30f0-48e9-b494-9b7d89323df2","Type":"ContainerStarted","Data":"d430318a5665df51b99a0f856ad2808976fe9ec9fae2f117c3223e746525296c"}
Oct 30 20:31:22.642250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.642223    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb" event={"ID":"a8a147ff-5aca-4876-952a-6c05cb21b6c4","Type":"ContainerStarted","Data":"54b17ff5145451bb39f03d83533662fb979760c3836172a9aa020b844eda5108"}
Oct 30 20:31:22.687781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:22.687702    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph" podStartSLOduration=1.687681593 podStartE2EDuration="1.687681593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:22.684633652 +0000 UTC m=+3407.185337377" watchObservedRunningTime="2026-10-30 20:31:22.687681593 +0000 UTC m=+3407.188385318"
Oct 30 20:31:23.497150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:23.497109    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"]
Oct 30 20:31:23.651459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:23.651414    4519 generic.go:356] "Generic (PLEG): container finished" podID="4633c0f2-30f0-48e9-b494-9b7d89323df2" containerID="15f0311e047aac9a2350e13a4ed50a8812ae1d918f07ec05e22799a89518a82e" exitCode=0
Oct 30 20:31:23.651691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:23.651496    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph" event={"ID":"4633c0f2-30f0-48e9-b494-9b7d89323df2","Type":"ContainerDied","Data":"15f0311e047aac9a2350e13a4ed50a8812ae1d918f07ec05e22799a89518a82e"}
Oct 30 20:31:23.653500 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:23.653456    4519 generic.go:356] "Generic (PLEG): container finished" podID="a8a147ff-5aca-4876-952a-6c05cb21b6c4" containerID="0f484e22994b9595d72c6fd0c4115bf6d2db5aca16e9b7cd89769dfb22cebb34" exitCode=0
Oct 30 20:31:23.653647 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:23.653506    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb" event={"ID":"a8a147ff-5aca-4876-952a-6c05cb21b6c4","Type":"ContainerDied","Data":"0f484e22994b9595d72c6fd0c4115bf6d2db5aca16e9b7cd89769dfb22cebb34"}
Oct 30 20:31:24.856658 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:24.856628    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:24.859981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:24.859957    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:25.028776 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.028731    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-openshift-ca\") pod \"4633c0f2-30f0-48e9-b494-9b7d89323df2\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") "
Oct 30 20:31:25.028997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.028786    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-xks-ca\") pod \"4633c0f2-30f0-48e9-b494-9b7d89323df2\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") "
Oct 30 20:31:25.028997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.028947    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-openshift-ca\") pod \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") "
Oct 30 20:31:25.028997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.028985    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-xks-ca\") pod \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") "
Oct 30 20:31:25.029191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.029017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4633c0f2-30f0-48e9-b494-9b7d89323df2-kube-api-access-tgskq\" (UniqueName: \"kubernetes.io/projected/4633c0f2-30f0-48e9-b494-9b7d89323df2-kube-api-access-tgskq\") pod \"4633c0f2-30f0-48e9-b494-9b7d89323df2\" (UID: \"4633c0f2-30f0-48e9-b494-9b7d89323df2\") "
Oct 30 20:31:25.029191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.029067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a147ff-5aca-4876-952a-6c05cb21b6c4-kube-api-access-s4pzz\" (UniqueName: \"kubernetes.io/projected/a8a147ff-5aca-4876-952a-6c05cb21b6c4-kube-api-access-s4pzz\") pod \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\" (UID: \"a8a147ff-5aca-4876-952a-6c05cb21b6c4\") "
Oct 30 20:31:25.029292 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.029183    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-openshift-ca" pod "4633c0f2-30f0-48e9-b494-9b7d89323df2" (UID: "4633c0f2-30f0-48e9-b494-9b7d89323df2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:25.029546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.029432    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:25.029693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.029635    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-openshift-ca" pod "a8a147ff-5aca-4876-952a-6c05cb21b6c4" (UID: "a8a147ff-5aca-4876-952a-6c05cb21b6c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:25.031388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.031359    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-xks-ca" pod "a8a147ff-5aca-4876-952a-6c05cb21b6c4" (UID: "a8a147ff-5aca-4876-952a-6c05cb21b6c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:25.031508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.031407    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a147ff-5aca-4876-952a-6c05cb21b6c4-kube-api-access-s4pzz" pod "a8a147ff-5aca-4876-952a-6c05cb21b6c4" (UID: "a8a147ff-5aca-4876-952a-6c05cb21b6c4"). InnerVolumeSpecName "kube-api-access-s4pzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:25.031680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.031660    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4633c0f2-30f0-48e9-b494-9b7d89323df2-kube-api-access-tgskq" pod "4633c0f2-30f0-48e9-b494-9b7d89323df2" (UID: "4633c0f2-30f0-48e9-b494-9b7d89323df2"). InnerVolumeSpecName "kube-api-access-tgskq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:25.031744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.031720    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-xks-ca" pod "4633c0f2-30f0-48e9-b494-9b7d89323df2" (UID: "4633c0f2-30f0-48e9-b494-9b7d89323df2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:25.130682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.130632    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:25.130682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.130671    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a147ff-5aca-4876-952a-6c05cb21b6c4-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:25.130682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.130683    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgskq\" (UniqueName: \"kubernetes.io/projected/4633c0f2-30f0-48e9-b494-9b7d89323df2-kube-api-access-tgskq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:25.130682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.130695    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4pzz\" (UniqueName: \"kubernetes.io/projected/a8a147ff-5aca-4876-952a-6c05cb21b6c4-kube-api-access-s4pzz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:25.130981 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.130705    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4633c0f2-30f0-48e9-b494-9b7d89323df2-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:25.670958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.670914    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"
Oct 30 20:31:25.670958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.670931    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph" event={"ID":"4633c0f2-30f0-48e9-b494-9b7d89323df2","Type":"ContainerDied","Data":"d430318a5665df51b99a0f856ad2808976fe9ec9fae2f117c3223e746525296c"}
Oct 30 20:31:25.671240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.670982    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d430318a5665df51b99a0f856ad2808976fe9ec9fae2f117c3223e746525296c"
Oct 30 20:31:25.673888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.673053    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb" event={"ID":"a8a147ff-5aca-4876-952a-6c05cb21b6c4","Type":"ContainerDied","Data":"54b17ff5145451bb39f03d83533662fb979760c3836172a9aa020b844eda5108"}
Oct 30 20:31:25.673888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.673099    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b17ff5145451bb39f03d83533662fb979760c3836172a9aa020b844eda5108"
Oct 30 20:31:25.673888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.673066    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"
Oct 30 20:31:25.963639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.963592    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"]
Oct 30 20:31:25.964164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.964139    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a8a147ff-5aca-4876-952a-6c05cb21b6c4" containerName="revoke-keys"
Oct 30 20:31:25.964164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.964161    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a147ff-5aca-4876-952a-6c05cb21b6c4" containerName="revoke-keys"
Oct 30 20:31:25.964299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.964195    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4633c0f2-30f0-48e9-b494-9b7d89323df2" containerName="revoke-keys"
Oct 30 20:31:25.964299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.964205    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4633c0f2-30f0-48e9-b494-9b7d89323df2" containerName="revoke-keys"
Oct 30 20:31:25.964375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.964321    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a147ff-5aca-4876-952a-6c05cb21b6c4" containerName="revoke-keys"
Oct 30 20:31:25.964375 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.964336    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4633c0f2-30f0-48e9-b494-9b7d89323df2" containerName="revoke-keys"
Oct 30 20:31:25.968024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.967998    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:25.981707 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.981671    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"]
Oct 30 20:31:25.986268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.986239    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:25.987579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.987553    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:25.987579 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:25.987556    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:31:26.000566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.000525    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"]
Oct 30 20:31:26.005268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.005236    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"]
Oct 30 20:31:26.033496 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.033231    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"]
Oct 30 20:31:26.038081 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.038042    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9cad9e6-54xxh"]
Oct 30 20:31:26.140878 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.140841    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckktr\" (UniqueName: \"kubernetes.io/projected/3994718d-da61-4ea1-9bed-2daa3939642a-kube-api-access-ckktr\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.141033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.140960    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.141033 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.141009    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.141111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.141046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.141111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.141078    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.141111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.141097    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hh8\" (UniqueName: \"kubernetes.io/projected/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-kube-api-access-89hh8\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.242334 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.242292    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckktr\" (UniqueName: \"kubernetes.io/projected/3994718d-da61-4ea1-9bed-2daa3939642a-kube-api-access-ckktr\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.242522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.242403    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.242522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.242430    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.243206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.242459    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.243329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.243253    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.243329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.243267    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.243329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.243312    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hh8\" (UniqueName: \"kubernetes.io/projected/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-kube-api-access-89hh8\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.244896 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.244535    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.249501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.247023    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.249501 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.247826    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.263832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.263774    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hh8\" (UniqueName: \"kubernetes.io/projected/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-kube-api-access-89hh8\") pod \"maas-api-revoke-sub-8d3b23c0-8hprg\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.263959 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.263881    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckktr\" (UniqueName: \"kubernetes.io/projected/3994718d-da61-4ea1-9bed-2daa3939642a-kube-api-access-ckktr\") pod \"maas-api-revoke-sub-0f6e5d57-82lc9\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.284459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.284428    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:26.321662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.321632    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:26.321792 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.321764    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344009    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6881d466-0536-4777-8072-0dc9d3733166-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6881d466-0536-4777-8072-0dc9d3733166-istiod-ca-cert\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344147    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6881d466-0536-4777-8072-0dc9d3733166-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6881d466-0536-4777-8072-0dc9d3733166-istio-podinfo\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344208    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-socket\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344252    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-kube-api-access-g6ndj\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-kube-api-access-g6ndj\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344291    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-credential-socket\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344329    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-data\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344368    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-istio-token\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-istio-token\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344404    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-envoy\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.344439    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-certs\") pod \"6881d466-0536-4777-8072-0dc9d3733166\" (UID: \"6881d466-0536-4777-8072-0dc9d3733166\") "
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.345119    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-certs" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:26.345855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.345539    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6881d466-0536-4777-8072-0dc9d3733166-istiod-ca-cert" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:26.347535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.347499    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-credential-socket" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:26.348083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.348055    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-data" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:26.348800 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.348750    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6881d466-0536-4777-8072-0dc9d3733166-istio-podinfo" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:26.349030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.349008    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-socket" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:26.350611 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.350577    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-envoy" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:26.351136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.351108    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-kube-api-access-g6ndj" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "kube-api-access-g6ndj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:26.351240 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.351150    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-istio-token" pod "6881d466-0536-4777-8072-0dc9d3733166" (UID: "6881d466-0536-4777-8072-0dc9d3733166"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:26.445911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445878    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6881d466-0536-4777-8072-0dc9d3733166-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.445911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445908    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.445911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445918    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6ndj\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-kube-api-access-g6ndj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.446148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445927    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.446148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445936    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.446148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445945    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6881d466-0536-4777-8072-0dc9d3733166-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.446148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445952    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.446148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445959    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6881d466-0536-4777-8072-0dc9d3733166-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.446148 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.445967    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6881d466-0536-4777-8072-0dc9d3733166-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:26.546801 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.546710    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"]
Oct 30 20:31:26.595436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.595403    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"]
Oct 30 20:31:26.596155 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:31:26.596129    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d48e57_30fa_43a6_8d31_cd1a4fec6b47.slice/crio-4c5447d51923a8bddda5fb75056e627d821878a3b0adea4afe9f82f5f520aeff WatchSource:0}: Error finding container 4c5447d51923a8bddda5fb75056e627d821878a3b0adea4afe9f82f5f520aeff: Status 404 returned error can't find the container with id 4c5447d51923a8bddda5fb75056e627d821878a3b0adea4afe9f82f5f520aeff
Oct 30 20:31:26.649914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.649888    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"]
Oct 30 20:31:26.653910 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:31:26.653879    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3994718d_da61_4ea1_9bed_2daa3939642a.slice/crio-e5845bb5d3258ab381b73b7b973eee8f6b99f767aeeb14b9319bb0b3d98ad910 WatchSource:0}: Error finding container e5845bb5d3258ab381b73b7b973eee8f6b99f767aeeb14b9319bb0b3d98ad910: Status 404 returned error can't find the container with id e5845bb5d3258ab381b73b7b973eee8f6b99f767aeeb14b9319bb0b3d98ad910
Oct 30 20:31:26.685102 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.685068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9" event={"ID":"3994718d-da61-4ea1-9bed-2daa3939642a","Type":"ContainerStarted","Data":"e5845bb5d3258ab381b73b7b973eee8f6b99f767aeeb14b9319bb0b3d98ad910"}
Oct 30 20:31:26.686783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.686755    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg" event={"ID":"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47","Type":"ContainerStarted","Data":"4c5447d51923a8bddda5fb75056e627d821878a3b0adea4afe9f82f5f520aeff"}
Oct 30 20:31:26.689153 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.689123    4519 generic.go:356] "Generic (PLEG): container finished" podID="6881d466-0536-4777-8072-0dc9d3733166" containerID="f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31" exitCode=0
Oct 30 20:31:26.689278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.689171    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"
Oct 30 20:31:26.689278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.689177    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" event={"ID":"6881d466-0536-4777-8072-0dc9d3733166","Type":"ContainerDied","Data":"f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31"}
Oct 30 20:31:26.689396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.689298    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g" event={"ID":"6881d466-0536-4777-8072-0dc9d3733166","Type":"ContainerDied","Data":"cee6b09d91ca226ad021b2666a264244d15a009f9e8fdd1f60dafeec5849f64b"}
Oct 30 20:31:26.689396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.689327    4519 scope.go:122] "RemoveContainer" containerID="f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31"
Oct 30 20:31:26.701660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.701639    4519 scope.go:122] "RemoveContainer" containerID="0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c"
Oct 30 20:31:26.711513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.711488    4519 scope.go:122] "RemoveContainer" containerID="f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31"
Oct 30 20:31:26.711805 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:26.711786    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31\": container with ID starting with f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31 not found: ID does not exist" containerID="f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31"
Oct 30 20:31:26.711859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.711814    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31"} err="failed to get container status \"f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31\": rpc error: code = NotFound desc = could not find container \"f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31\": container with ID starting with f1201c12f5564c0476a28ab4fd1404a3cc09b3df3184f062a845a398b86dcb31 not found: ID does not exist"
Oct 30 20:31:26.711859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.711831    4519 scope.go:122] "RemoveContainer" containerID="0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c"
Oct 30 20:31:26.712081 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:26.712058    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c\": container with ID starting with 0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c not found: ID does not exist" containerID="0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c"
Oct 30 20:31:26.712128 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.712088    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c"} err="failed to get container status \"0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c\": rpc error: code = NotFound desc = could not find container \"0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c\": container with ID starting with 0232891c3b576c8b2cb4a568bda9cb2e068cd87465aac8cbbaee641c5baa628c not found: ID does not exist"
Oct 30 20:31:26.744027 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.743948    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"]
Oct 30 20:31:26.747854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:26.747822    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-27a4d3-openshift-default-54ff49cc9f-2h79g"]
Oct 30 20:31:27.612828 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:27.612792    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db76303-5059-4795-8a31-6f05e25aea64" path="/var/lib/kubelet/pods/1db76303-5059-4795-8a31-6f05e25aea64/volumes"
Oct 30 20:31:27.613310 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:27.613294    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6881d466-0536-4777-8072-0dc9d3733166" path="/var/lib/kubelet/pods/6881d466-0536-4777-8072-0dc9d3733166/volumes"
Oct 30 20:31:27.700819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:27.700782    4519 generic.go:356] "Generic (PLEG): container finished" podID="3994718d-da61-4ea1-9bed-2daa3939642a" containerID="295b898777ac6c261e23ebca58603ba69fa1c92d9de300be16d1c01217041b64" exitCode=0
Oct 30 20:31:27.701040 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:27.700869    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9" event={"ID":"3994718d-da61-4ea1-9bed-2daa3939642a","Type":"ContainerDied","Data":"295b898777ac6c261e23ebca58603ba69fa1c92d9de300be16d1c01217041b64"}
Oct 30 20:31:27.702899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:27.702870    4519 generic.go:356] "Generic (PLEG): container finished" podID="c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" containerID="710499f8eba5d4a85788a541f60e6b3f2432fd747b4a695cc860d40c844bb865" exitCode=0
Oct 30 20:31:27.703049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:27.702925    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg" event={"ID":"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47","Type":"ContainerDied","Data":"710499f8eba5d4a85788a541f60e6b3f2432fd747b4a695cc860d40c844bb865"}
Oct 30 20:31:28.497702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.497650    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" podUID="dae99e72-5490-4883-be29-e73f0f85d920" containerName="payload-pre-processing" containerID="cri-o://02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69" gracePeriod=25
Oct 30 20:31:28.674199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.674160    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main" containerID="cri-o://371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173" gracePeriod=45
Oct 30 20:31:28.882384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.882321    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:28.885314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.885290    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:28.968684 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.968650    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 20:31:28.973737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.973709    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3994718d-da61-4ea1-9bed-2daa3939642a-kube-api-access-ckktr\" (UniqueName: \"kubernetes.io/projected/3994718d-da61-4ea1-9bed-2daa3939642a-kube-api-access-ckktr\") pod \"3994718d-da61-4ea1-9bed-2daa3939642a\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") "
Oct 30 20:31:28.973891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.973758    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-xks-ca\") pod \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") "
Oct 30 20:31:28.973891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.973800    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-kube-api-access-89hh8\" (UniqueName: \"kubernetes.io/projected/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-kube-api-access-89hh8\") pod \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") "
Oct 30 20:31:28.973891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.973875    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-xks-ca\") pod \"3994718d-da61-4ea1-9bed-2daa3939642a\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") "
Oct 30 20:31:28.974086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.973971    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-openshift-ca\") pod \"3994718d-da61-4ea1-9bed-2daa3939642a\" (UID: \"3994718d-da61-4ea1-9bed-2daa3939642a\") "
Oct 30 20:31:28.974086 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.974051    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-openshift-ca\") pod \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\" (UID: \"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47\") "
Oct 30 20:31:28.974425 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.974392    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-openshift-ca" pod "3994718d-da61-4ea1-9bed-2daa3939642a" (UID: "3994718d-da61-4ea1-9bed-2daa3939642a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:28.974983 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.974903    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-openshift-ca" pod "c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" (UID: "c7d48e57-30fa-43a6-8d31-cd1a4fec6b47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:28.976241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.976215    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-kube-api-access-89hh8" pod "c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" (UID: "c7d48e57-30fa-43a6-8d31-cd1a4fec6b47"). InnerVolumeSpecName "kube-api-access-89hh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:28.976397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.976380    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-xks-ca" pod "c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" (UID: "c7d48e57-30fa-43a6-8d31-cd1a4fec6b47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:28.976532 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.976509    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-xks-ca" pod "3994718d-da61-4ea1-9bed-2daa3939642a" (UID: "3994718d-da61-4ea1-9bed-2daa3939642a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:28.978179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:28.978152    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3994718d-da61-4ea1-9bed-2daa3939642a-kube-api-access-ckktr" pod "3994718d-da61-4ea1-9bed-2daa3939642a" (UID: "3994718d-da61-4ea1-9bed-2daa3939642a"). InnerVolumeSpecName "kube-api-access-ckktr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:29.075103 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075013    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-kube-api-access-7v6db\" (UniqueName: \"kubernetes.io/projected/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-kube-api-access-7v6db\") pod \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") "
Oct 30 20:31:29.075276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075105    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-dshm\") pod \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") "
Oct 30 20:31:29.075276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075216    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-model-cache\") pod \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") "
Oct 30 20:31:29.075276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075271    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tls-certs\") pod \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") "
Oct 30 20:31:29.075505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075316    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-home\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-home\") pod \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") "
Oct 30 20:31:29.075505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075348    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tmp-dir\") pod \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\" (UID: \"a39f4406-705a-4fc3-9f5d-46c2cfd750f8\") "
Oct 30 20:31:29.075647 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075507    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-model-cache" pod "a39f4406-705a-4fc3-9f5d-46c2cfd750f8" (UID: "a39f4406-705a-4fc3-9f5d-46c2cfd750f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:29.075733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075714    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.075802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075738    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckktr\" (UniqueName: \"kubernetes.io/projected/3994718d-da61-4ea1-9bed-2daa3939642a-kube-api-access-ckktr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.075802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075756    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.075802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075745    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-home" pod "a39f4406-705a-4fc3-9f5d-46c2cfd750f8" (UID: "a39f4406-705a-4fc3-9f5d-46c2cfd750f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:29.075802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075772    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89hh8\" (UniqueName: \"kubernetes.io/projected/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47-kube-api-access-89hh8\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.075963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075793    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tmp-dir" pod "a39f4406-705a-4fc3-9f5d-46c2cfd750f8" (UID: "a39f4406-705a-4fc3-9f5d-46c2cfd750f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:29.075963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075814    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.075963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075833    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.075963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.075849    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994718d-da61-4ea1-9bed-2daa3939642a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.077954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.077915    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tls-certs" pod "a39f4406-705a-4fc3-9f5d-46c2cfd750f8" (UID: "a39f4406-705a-4fc3-9f5d-46c2cfd750f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:29.078111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.078087    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-dshm" pod "a39f4406-705a-4fc3-9f5d-46c2cfd750f8" (UID: "a39f4406-705a-4fc3-9f5d-46c2cfd750f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:29.078225 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.078207    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-kube-api-access-7v6db" pod "a39f4406-705a-4fc3-9f5d-46c2cfd750f8" (UID: "a39f4406-705a-4fc3-9f5d-46c2cfd750f8"). InnerVolumeSpecName "kube-api-access-7v6db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:29.176946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.176906    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.176946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.176939    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.176946 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.176949    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.177193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.176956    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.177193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.176966    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6db\" (UniqueName: \"kubernetes.io/projected/a39f4406-705a-4fc3-9f5d-46c2cfd750f8-kube-api-access-7v6db\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:29.719519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.719462    4519 generic.go:356] "Generic (PLEG): container finished" podID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerID="371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173" exitCode=0
Oct 30 20:31:29.720024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.719569    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"
Oct 30 20:31:29.720024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.719568    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" event={"ID":"a39f4406-705a-4fc3-9f5d-46c2cfd750f8","Type":"ContainerDied","Data":"371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173"}
Oct 30 20:31:29.720024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.719693    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft" event={"ID":"a39f4406-705a-4fc3-9f5d-46c2cfd750f8","Type":"ContainerDied","Data":"b112e5a3471e6286ee547248312cd50cbb3d3a7ea59757bf4468a3a382d3c66b"}
Oct 30 20:31:29.720024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.719714    4519 scope.go:122] "RemoveContainer" containerID="371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173"
Oct 30 20:31:29.721560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.721442    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg" event={"ID":"c7d48e57-30fa-43a6-8d31-cd1a4fec6b47","Type":"ContainerDied","Data":"4c5447d51923a8bddda5fb75056e627d821878a3b0adea4afe9f82f5f520aeff"}
Oct 30 20:31:29.721560 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.721548    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c5447d51923a8bddda5fb75056e627d821878a3b0adea4afe9f82f5f520aeff"
Oct 30 20:31:29.721811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.721676    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"
Oct 30 20:31:29.723369 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.723351    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"
Oct 30 20:31:29.723369 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.723360    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9" event={"ID":"3994718d-da61-4ea1-9bed-2daa3939642a","Type":"ContainerDied","Data":"e5845bb5d3258ab381b73b7b973eee8f6b99f767aeeb14b9319bb0b3d98ad910"}
Oct 30 20:31:29.723570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.723403    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5845bb5d3258ab381b73b7b973eee8f6b99f767aeeb14b9319bb0b3d98ad910"
Oct 30 20:31:29.739573 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.739553    4519 scope.go:122] "RemoveContainer" containerID="371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173"
Oct 30 20:31:29.739887 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:29.739862    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173\": container with ID starting with 371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173 not found: ID does not exist" containerID="371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173"
Oct 30 20:31:29.739970 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.739895    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173"} err="failed to get container status \"371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173\": rpc error: code = NotFound desc = could not find container \"371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173\": container with ID starting with 371db2d6f5c51dfe8339b752f9090ab3231e31d7caaccbf821e81471bfdf2173 not found: ID does not exist"
Oct 30 20:31:29.784723 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.784691    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"]
Oct 30 20:31:29.787848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:29.787820    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-2-w0-5db436-kserve-859b87564d-cmjft"]
Oct 30 20:31:30.428876 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:30.428839    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" podUID="dae99e72-5490-4883-be29-e73f0f85d920" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:31:31.611594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:31.611556    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" path="/var/lib/kubelet/pods/a39f4406-705a-4fc3-9f5d-46c2cfd750f8/volumes"
Oct 30 20:31:32.221328 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:32.221286    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:31:33.093058 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.093017    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main" containerID="cri-o://3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44" gracePeriod=45
Oct 30 20:31:33.353599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.353536    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 20:31:33.416300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416260    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-dshm\") pod \"8232473a-8c1a-4f51-80d4-578eee5bc87c\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") "
Oct 30 20:31:33.416300 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416302    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-model-cache\") pod \"8232473a-8c1a-4f51-80d4-578eee5bc87c\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") "
Oct 30 20:31:33.416596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416369    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-tmp-dir\") pod \"8232473a-8c1a-4f51-80d4-578eee5bc87c\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") "
Oct 30 20:31:33.416596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416399    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-home\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-home\") pod \"8232473a-8c1a-4f51-80d4-578eee5bc87c\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") "
Oct 30 20:31:33.416596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416421    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8232473a-8c1a-4f51-80d4-578eee5bc87c-kube-api-access-jdlkr\" (UniqueName: \"kubernetes.io/projected/8232473a-8c1a-4f51-80d4-578eee5bc87c-kube-api-access-jdlkr\") pod \"8232473a-8c1a-4f51-80d4-578eee5bc87c\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") "
Oct 30 20:31:33.416596 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416447    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8232473a-8c1a-4f51-80d4-578eee5bc87c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8232473a-8c1a-4f51-80d4-578eee5bc87c-tls-certs\") pod \"8232473a-8c1a-4f51-80d4-578eee5bc87c\" (UID: \"8232473a-8c1a-4f51-80d4-578eee5bc87c\") "
Oct 30 20:31:33.416838 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416721    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-model-cache" pod "8232473a-8c1a-4f51-80d4-578eee5bc87c" (UID: "8232473a-8c1a-4f51-80d4-578eee5bc87c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:33.417021 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.416995    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:33.417449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.417420    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-tmp-dir" pod "8232473a-8c1a-4f51-80d4-578eee5bc87c" (UID: "8232473a-8c1a-4f51-80d4-578eee5bc87c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:33.417604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.417447    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-home" pod "8232473a-8c1a-4f51-80d4-578eee5bc87c" (UID: "8232473a-8c1a-4f51-80d4-578eee5bc87c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:33.419109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.419075    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8232473a-8c1a-4f51-80d4-578eee5bc87c-kube-api-access-jdlkr" pod "8232473a-8c1a-4f51-80d4-578eee5bc87c" (UID: "8232473a-8c1a-4f51-80d4-578eee5bc87c"). InnerVolumeSpecName "kube-api-access-jdlkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:33.419230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.419213    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-dshm" pod "8232473a-8c1a-4f51-80d4-578eee5bc87c" (UID: "8232473a-8c1a-4f51-80d4-578eee5bc87c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:33.419289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.419230    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8232473a-8c1a-4f51-80d4-578eee5bc87c-tls-certs" pod "8232473a-8c1a-4f51-80d4-578eee5bc87c" (UID: "8232473a-8c1a-4f51-80d4-578eee5bc87c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:33.518241 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.518197    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:33.518395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.518262    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlkr\" (UniqueName: \"kubernetes.io/projected/8232473a-8c1a-4f51-80d4-578eee5bc87c-kube-api-access-jdlkr\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:33.518395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.518281    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8232473a-8c1a-4f51-80d4-578eee5bc87c-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:33.518395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.518295    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:33.518395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.518336    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8232473a-8c1a-4f51-80d4-578eee5bc87c-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:33.760602 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.760562    4519 generic.go:356] "Generic (PLEG): container finished" podID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerID="3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44" exitCode=0
Oct 30 20:31:33.760823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.760636    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" event={"ID":"8232473a-8c1a-4f51-80d4-578eee5bc87c","Type":"ContainerDied","Data":"3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44"}
Oct 30 20:31:33.760823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.760669    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk" event={"ID":"8232473a-8c1a-4f51-80d4-578eee5bc87c","Type":"ContainerDied","Data":"bd1d4b957666b3bc9db3e19f42c742a78696d82f423010e18065e095709f2d69"}
Oct 30 20:31:33.760823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.760687    4519 scope.go:122] "RemoveContainer" containerID="3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44"
Oct 30 20:31:33.760823 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.760640    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"
Oct 30 20:31:33.771813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.771792    4519 scope.go:122] "RemoveContainer" containerID="3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44"
Oct 30 20:31:33.772111 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:33.772084    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44\": container with ID starting with 3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44 not found: ID does not exist" containerID="3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44"
Oct 30 20:31:33.772224 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.772121    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44"} err="failed to get container status \"3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44\": rpc error: code = NotFound desc = could not find container \"3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44\": container with ID starting with 3003500590b21ecb605491ac03487c3e95022989ed644f038f204e62124e2d44 not found: ID does not exist"
Oct 30 20:31:33.812129 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.812097    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"]
Oct 30 20:31:33.825566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:33.825539    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-2-w1-109cfe-kserve-68684f46b7-px7kk"]
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.307409    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"]
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308100    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308119    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308184    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" containerName="revoke-keys"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308192    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" containerName="revoke-keys"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308209    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308218    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308237    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3994718d-da61-4ea1-9bed-2daa3939642a" containerName="revoke-keys"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308245    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3994718d-da61-4ea1-9bed-2daa3939642a" containerName="revoke-keys"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308260    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308269    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308351    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308366    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" containerName="revoke-keys"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308377    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" containerName="main"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308386    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3994718d-da61-4ea1-9bed-2daa3939642a" containerName="revoke-keys"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.308394    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39f4406-705a-4fc3-9f5d-46c2cfd750f8" containerName="main"
Oct 30 20:31:35.319926 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.316886    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.338373 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.332034    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"]
Oct 30 20:31:35.339299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.339254    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:35.339420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.339313    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:31:35.441230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.441189    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.441230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.441233    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5fz\" (UniqueName: \"kubernetes.io/projected/cbe26dba-8ed3-4ea9-8f62-cf6829667465-kube-api-access-gt5fz\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.441524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.441333    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.543047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.543008    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.543232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.543148    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.543232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.543184    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5fz\" (UniqueName: \"kubernetes.io/projected/cbe26dba-8ed3-4ea9-8f62-cf6829667465-kube-api-access-gt5fz\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.543844 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.543813    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.545702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.545677    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.562337 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.562304    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5fz\" (UniqueName: \"kubernetes.io/projected/cbe26dba-8ed3-4ea9-8f62-cf6829667465-kube-api-access-gt5fz\") pod \"maas-api-revoke-sub-e30a5110-btmm9\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.611897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.611814    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8232473a-8c1a-4f51-80d4-578eee5bc87c" path="/var/lib/kubelet/pods/8232473a-8c1a-4f51-80d4-578eee5bc87c/volumes"
Oct 30 20:31:35.625863 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.625830    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"]
Oct 30 20:31:35.632799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.632760    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5db436-btbzt"]
Oct 30 20:31:35.646149 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.646120    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:35.957238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:35.957204    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"]
Oct 30 20:31:35.958352 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:31:35.958318    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe26dba_8ed3_4ea9_8f62_cf6829667465.slice/crio-fe8e7987bdb4c9d76e11ae95c82fd916b737a207445ca148d315fb4372c629b8 WatchSource:0}: Error finding container fe8e7987bdb4c9d76e11ae95c82fd916b737a207445ca148d315fb4372c629b8: Status 404 returned error can't find the container with id fe8e7987bdb4c9d76e11ae95c82fd916b737a207445ca148d315fb4372c629b8
Oct 30 20:31:36.788867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:36.788831    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9" event={"ID":"cbe26dba-8ed3-4ea9-8f62-cf6829667465","Type":"ContainerStarted","Data":"35411345bbfd2e77f5382335fb61350997cd020b9ffbe2cde4fb5aab15b43718"}
Oct 30 20:31:36.789264 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:36.788898    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9" event={"ID":"cbe26dba-8ed3-4ea9-8f62-cf6829667465","Type":"ContainerStarted","Data":"fe8e7987bdb4c9d76e11ae95c82fd916b737a207445ca148d315fb4372c629b8"}
Oct 30 20:31:36.828057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:36.828003    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9" podStartSLOduration=1.8279900850000002 podStartE2EDuration="1.827990085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:36.827309732 +0000 UTC m=+3421.328013460" watchObservedRunningTime="2026-10-30 20:31:36.827990085 +0000 UTC m=+3421.328693809"
Oct 30 20:31:37.611535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:37.611498    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4caf50-3b35-4b94-a892-f56ca5e9f1c4" path="/var/lib/kubelet/pods/4b4caf50-3b35-4b94-a892-f56ca5e9f1c4/volumes"
Oct 30 20:31:37.787053 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:37.787012    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"]
Oct 30 20:31:37.800691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:37.800650    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-109cfe-mns9z"]
Oct 30 20:31:37.802495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:37.802438    4519 generic.go:356] "Generic (PLEG): container finished" podID="cbe26dba-8ed3-4ea9-8f62-cf6829667465" containerID="35411345bbfd2e77f5382335fb61350997cd020b9ffbe2cde4fb5aab15b43718" exitCode=0
Oct 30 20:31:37.802618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:37.802511    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9" event={"ID":"cbe26dba-8ed3-4ea9-8f62-cf6829667465","Type":"ContainerDied","Data":"35411345bbfd2e77f5382335fb61350997cd020b9ffbe2cde4fb5aab15b43718"}
Oct 30 20:31:38.960824 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:38.960787    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:39.080932 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.080826    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-xks-ca\") pod \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") "
Oct 30 20:31:39.081111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.080956    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe26dba-8ed3-4ea9-8f62-cf6829667465-kube-api-access-gt5fz\" (UniqueName: \"kubernetes.io/projected/cbe26dba-8ed3-4ea9-8f62-cf6829667465-kube-api-access-gt5fz\") pod \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") "
Oct 30 20:31:39.081111 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.081067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-openshift-ca\") pod \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\" (UID: \"cbe26dba-8ed3-4ea9-8f62-cf6829667465\") "
Oct 30 20:31:39.081451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.081419    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-openshift-ca" pod "cbe26dba-8ed3-4ea9-8f62-cf6829667465" (UID: "cbe26dba-8ed3-4ea9-8f62-cf6829667465"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:39.083449 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.083421    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe26dba-8ed3-4ea9-8f62-cf6829667465-kube-api-access-gt5fz" pod "cbe26dba-8ed3-4ea9-8f62-cf6829667465" (UID: "cbe26dba-8ed3-4ea9-8f62-cf6829667465"). InnerVolumeSpecName "kube-api-access-gt5fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:39.083595 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.083498    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-xks-ca" pod "cbe26dba-8ed3-4ea9-8f62-cf6829667465" (UID: "cbe26dba-8ed3-4ea9-8f62-cf6829667465"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:39.182431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.182393    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:39.182431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.182426    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe26dba-8ed3-4ea9-8f62-cf6829667465-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:39.182431 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.182437    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5fz\" (UniqueName: \"kubernetes.io/projected/cbe26dba-8ed3-4ea9-8f62-cf6829667465-kube-api-access-gt5fz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:39.611648 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.611600    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8301f28f-d413-45c6-9a56-3478f794c2ba" path="/var/lib/kubelet/pods/8301f28f-d413-45c6-9a56-3478f794c2ba/volumes"
Oct 30 20:31:39.822458 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.822415    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9" event={"ID":"cbe26dba-8ed3-4ea9-8f62-cf6829667465","Type":"ContainerDied","Data":"fe8e7987bdb4c9d76e11ae95c82fd916b737a207445ca148d315fb4372c629b8"}
Oct 30 20:31:39.822660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.822516    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe8e7987bdb4c9d76e11ae95c82fd916b737a207445ca148d315fb4372c629b8"
Oct 30 20:31:39.822660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:39.822424    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"
Oct 30 20:31:40.428453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:40.428414    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" podUID="dae99e72-5490-4883-be29-e73f0f85d920" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:31:41.568938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.568885    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main" containerID="cri-o://cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c" gracePeriod=45
Oct 30 20:31:41.821006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.820947    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 20:31:41.839568 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.839539    4519 generic.go:356] "Generic (PLEG): container finished" podID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerID="cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c" exitCode=0
Oct 30 20:31:41.839733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.839577    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"
Oct 30 20:31:41.839733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.839613    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" event={"ID":"77ff036e-01cc-42be-8059-e4721ce56fbc","Type":"ContainerDied","Data":"cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c"}
Oct 30 20:31:41.839733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.839664    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm" event={"ID":"77ff036e-01cc-42be-8059-e4721ce56fbc","Type":"ContainerDied","Data":"19cf5db916698a079f3ba2e77474c43209c39b4e294736a01ca9f0f7a122d921"}
Oct 30 20:31:41.839733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.839686    4519 scope.go:122] "RemoveContainer" containerID="cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c"
Oct 30 20:31:41.857167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.857137    4519 scope.go:122] "RemoveContainer" containerID="cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c"
Oct 30 20:31:41.857597 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:41.857561    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c\": container with ID starting with cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c not found: ID does not exist" containerID="cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c"
Oct 30 20:31:41.857680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.857616    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c"} err="failed to get container status \"cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c\": rpc error: code = NotFound desc = could not find container \"cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c\": container with ID starting with cf6e27935a8f9f025adaf4287df40c84ee7ec1198f6e5154b7541245c8ed412c not found: ID does not exist"
Oct 30 20:31:41.907638 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.907587    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ff036e-01cc-42be-8059-e4721ce56fbc-kube-api-access-74krn\" (UniqueName: \"kubernetes.io/projected/77ff036e-01cc-42be-8059-e4721ce56fbc-kube-api-access-74krn\") pod \"77ff036e-01cc-42be-8059-e4721ce56fbc\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") "
Oct 30 20:31:41.907913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.907652    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-model-cache\") pod \"77ff036e-01cc-42be-8059-e4721ce56fbc\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") "
Oct 30 20:31:41.907913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.907848    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-dshm\") pod \"77ff036e-01cc-42be-8059-e4721ce56fbc\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") "
Oct 30 20:31:41.907913 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.907888    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-home\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-home\") pod \"77ff036e-01cc-42be-8059-e4721ce56fbc\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") "
Oct 30 20:31:41.908115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.907970    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ff036e-01cc-42be-8059-e4721ce56fbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/77ff036e-01cc-42be-8059-e4721ce56fbc-tls-certs\") pod \"77ff036e-01cc-42be-8059-e4721ce56fbc\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") "
Oct 30 20:31:41.908115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.907967    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-model-cache" pod "77ff036e-01cc-42be-8059-e4721ce56fbc" (UID: "77ff036e-01cc-42be-8059-e4721ce56fbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:41.908115 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.908075    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-tmp-dir\") pod \"77ff036e-01cc-42be-8059-e4721ce56fbc\" (UID: \"77ff036e-01cc-42be-8059-e4721ce56fbc\") "
Oct 30 20:31:41.908380 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.908197    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-home" pod "77ff036e-01cc-42be-8059-e4721ce56fbc" (UID: "77ff036e-01cc-42be-8059-e4721ce56fbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:41.908450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.908398    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-tmp-dir" pod "77ff036e-01cc-42be-8059-e4721ce56fbc" (UID: "77ff036e-01cc-42be-8059-e4721ce56fbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:41.908749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.908721    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:41.908855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.908753    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:41.908855 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.908768    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:41.910712 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.910686    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ff036e-01cc-42be-8059-e4721ce56fbc-tls-certs" pod "77ff036e-01cc-42be-8059-e4721ce56fbc" (UID: "77ff036e-01cc-42be-8059-e4721ce56fbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:41.910819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.910751    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ff036e-01cc-42be-8059-e4721ce56fbc-kube-api-access-74krn" pod "77ff036e-01cc-42be-8059-e4721ce56fbc" (UID: "77ff036e-01cc-42be-8059-e4721ce56fbc"). InnerVolumeSpecName "kube-api-access-74krn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:41.910870 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:41.910842    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-dshm" pod "77ff036e-01cc-42be-8059-e4721ce56fbc" (UID: "77ff036e-01cc-42be-8059-e4721ce56fbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:42.010193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:42.010157    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74krn\" (UniqueName: \"kubernetes.io/projected/77ff036e-01cc-42be-8059-e4721ce56fbc-kube-api-access-74krn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:42.010193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:42.010190    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ff036e-01cc-42be-8059-e4721ce56fbc-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:42.010193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:42.010201    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ff036e-01cc-42be-8059-e4721ce56fbc-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:42.185206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:42.185165    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"]
Oct 30 20:31:42.197232 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:42.197200    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/simulator-w6-f349a1-kserve-75864459cf-4fddm"]
Oct 30 20:31:43.613212 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:43.613174    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" path="/var/lib/kubelet/pods/77ff036e-01cc-42be-8059-e4721ce56fbc/volumes"
Oct 30 20:31:45.557937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.557902    4519 scope.go:122] "RemoveContainer" containerID="2d31ef180fc2b58f796bb177704c08f2d2d2159558cfe64d10bb9d256befff86"
Oct 30 20:31:45.570509 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.570485    4519 scope.go:122] "RemoveContainer" containerID="51bf4039654de26a597de7067c7428a688504fea47bd5d520919fd6536e4cad3"
Oct 30 20:31:45.582997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.582969    4519 scope.go:122] "RemoveContainer" containerID="9700f8c7f2a2ef22ccf39c1a1e091c6bafe0ed19eca6d8e631598db708767693"
Oct 30 20:31:45.592980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.592947    4519 scope.go:122] "RemoveContainer" containerID="8d1910a99e5f0a398e91fa5a0bec8b8363b16a7cec10c99d1bf60455e88e2403"
Oct 30 20:31:45.603504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.603476    4519 scope.go:122] "RemoveContainer" containerID="cc7ed579e71b7c744d4bb010b46466c6610d7a56695227dc7078890b4066c928"
Oct 30 20:31:45.613436 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.613412    4519 scope.go:122] "RemoveContainer" containerID="73eeb6a81614e48bd378c527ec04b09703a697978a174d02c48ef96385825b80"
Oct 30 20:31:45.623048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.623028    4519 scope.go:122] "RemoveContainer" containerID="344d7bde428a368ad656d183eec0fd4345db715bd4df18a2351eaf9a5cd0e6b4"
Oct 30 20:31:45.636104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:45.635525    4519 scope.go:122] "RemoveContainer" containerID="ff663d81dce5ddf829025d656168fdcef41f36f06c2f85397e64176b9efc3800"
Oct 30 20:31:48.745435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.745411    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 20:31:48.880455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.880375    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae99e72-5490-4883-be29-e73f0f85d920-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae99e72-5490-4883-be29-e73f0f85d920-plugins-config-volume\") pod \"dae99e72-5490-4883-be29-e73f0f85d920\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") "
Oct 30 20:31:48.880455 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.880416    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae99e72-5490-4883-be29-e73f0f85d920-kube-api-access-flpld\" (UniqueName: \"kubernetes.io/projected/dae99e72-5490-4883-be29-e73f0f85d920-kube-api-access-flpld\") pod \"dae99e72-5490-4883-be29-e73f0f85d920\" (UID: \"dae99e72-5490-4883-be29-e73f0f85d920\") "
Oct 30 20:31:48.880798 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.880774    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae99e72-5490-4883-be29-e73f0f85d920-plugins-config-volume" pod "dae99e72-5490-4883-be29-e73f0f85d920" (UID: "dae99e72-5490-4883-be29-e73f0f85d920"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:48.880891 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.880868    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae99e72-5490-4883-be29-e73f0f85d920-plugins-config-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:48.882614 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.882581    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae99e72-5490-4883-be29-e73f0f85d920-kube-api-access-flpld" pod "dae99e72-5490-4883-be29-e73f0f85d920" (UID: "dae99e72-5490-4883-be29-e73f0f85d920"). InnerVolumeSpecName "kube-api-access-flpld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:48.897262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.897235    4519 generic.go:356] "Generic (PLEG): container finished" podID="dae99e72-5490-4883-be29-e73f0f85d920" containerID="02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69" exitCode=0
Oct 30 20:31:48.897386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.897273    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"
Oct 30 20:31:48.897386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.897311    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" event={"ID":"dae99e72-5490-4883-be29-e73f0f85d920","Type":"ContainerDied","Data":"02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69"}
Oct 30 20:31:48.897386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.897362    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz" event={"ID":"dae99e72-5490-4883-be29-e73f0f85d920","Type":"ContainerDied","Data":"094095b4a57994a99ba9979e0f66c2ac1d59ea35848ce7aecce358e8b981e60d"}
Oct 30 20:31:48.897550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.897387    4519 scope.go:122] "RemoveContainer" containerID="02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69"
Oct 30 20:31:48.907385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.907365    4519 scope.go:122] "RemoveContainer" containerID="02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69"
Oct 30 20:31:48.907681 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:31:48.907663    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69\": container with ID starting with 02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69 not found: ID does not exist" containerID="02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69"
Oct 30 20:31:48.907747 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.907690    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69"} err="failed to get container status \"02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69\": rpc error: code = NotFound desc = could not find container \"02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69\": container with ID starting with 02a32d9ec44183bea822519f30f7cdfa109e4455491fe0af7896aa7eac44fc69 not found: ID does not exist"
Oct 30 20:31:48.950330 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.950303    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"]
Oct 30 20:31:48.957694 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.957662    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-109cfe-6bd7f8f6ff-s4cxz"]
Oct 30 20:31:48.982018 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:48.981988    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flpld\" (UniqueName: \"kubernetes.io/projected/dae99e72-5490-4883-be29-e73f0f85d920-kube-api-access-flpld\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:31:49.611271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:49.611234    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae99e72-5490-4883-be29-e73f0f85d920" path="/var/lib/kubelet/pods/dae99e72-5490-4883-be29-e73f0f85d920/volumes"
Oct 30 20:31:57.976519 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:57.976458    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"]
Oct 30 20:31:57.977593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:31:57.977531    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy" containerID="cri-o://4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2" gracePeriod=30
Oct 30 20:32:02.220508 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:32:02.220445    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:32:03.252263 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.252181    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-xbjnv"]
Oct 30 20:32:03.252749 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.252531    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerName="manager" containerID="cri-o://2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180" gracePeriod=30
Oct 30 20:32:03.379833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.379809    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.423928    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-kube-api-access-hstz9\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-kube-api-access-hstz9\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424035    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istiod-ca-cert\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424068    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-certs\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424151    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-envoy\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424209    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-credential-socket\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424242    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-token\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-token\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424282    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-data\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424350    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-podinfo\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.424423    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-socket\") pod \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\" (UID: \"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc\") "
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.425113    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-socket" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.426247    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istiod-ca-cert" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:32:03.427640 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.426531    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-certs" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:03.429088 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.429052    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-data" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:03.429408 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.429389    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-credential-socket" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:03.433818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.433690    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-kube-api-access-hstz9" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "kube-api-access-hstz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:03.435521 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.433928    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-envoy" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:03.440182 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.440142    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-token" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:03.440887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.440842    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-podinfo" pod "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" (UID: "ceb3f0f5-e299-4472-aa62-6936c4cf8bbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:32:03.515010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.514978    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 20:32:03.525927 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.525889    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-kube-api-access-96w6v\" (UniqueName: \"kubernetes.io/projected/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-kube-api-access-96w6v\") pod \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") "
Oct 30 20:32:03.526118 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526034    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-webhook-cert\") pod \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") "
Oct 30 20:32:03.526193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526125    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-metrics-certs\") pod \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\" (UID: \"ae9324bb-be21-4f9f-a5cd-a983a2d585aa\") "
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526353    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstz9\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-kube-api-access-hstz9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526374    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526390    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526404    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526418    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526431    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526444    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526456    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.526645 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.526484    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.528210 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.528180    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-kube-api-access-96w6v" pod "ae9324bb-be21-4f9f-a5cd-a983a2d585aa" (UID: "ae9324bb-be21-4f9f-a5cd-a983a2d585aa"). InnerVolumeSpecName "kube-api-access-96w6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:03.528693 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.528665    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-webhook-cert" pod "ae9324bb-be21-4f9f-a5cd-a983a2d585aa" (UID: "ae9324bb-be21-4f9f-a5cd-a983a2d585aa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:03.528807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.528701    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-metrics-certs" pod "ae9324bb-be21-4f9f-a5cd-a983a2d585aa" (UID: "ae9324bb-be21-4f9f-a5cd-a983a2d585aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:03.627140 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.627111    4519 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-webhook-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.627140 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.627138    4519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-metrics-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:03.627346 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:03.627149    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96w6v\" (UniqueName: \"kubernetes.io/projected/ae9324bb-be21-4f9f-a5cd-a983a2d585aa-kube-api-access-96w6v\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:04.022735 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.022702    4519 generic.go:356] "Generic (PLEG): container finished" podID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerID="2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180" exitCode=0
Oct 30 20:32:04.022893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.022745    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-xbjnv"
Oct 30 20:32:04.022893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.022781    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" event={"ID":"ae9324bb-be21-4f9f-a5cd-a983a2d585aa","Type":"ContainerDied","Data":"2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180"}
Oct 30 20:32:04.022893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.022839    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-xbjnv" event={"ID":"ae9324bb-be21-4f9f-a5cd-a983a2d585aa","Type":"ContainerDied","Data":"7035bcb64c049a38628b5449a5abadb7c2f96e6cd6f8b044a5cb6003e7305d61"}
Oct 30 20:32:04.022893 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.022867    4519 scope.go:122] "RemoveContainer" containerID="2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180"
Oct 30 20:32:04.024538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.024512    4519 generic.go:356] "Generic (PLEG): container finished" podID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerID="4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2" exitCode=0
Oct 30 20:32:04.024696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.024564    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"
Oct 30 20:32:04.024696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.024596    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" event={"ID":"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc","Type":"ContainerDied","Data":"4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2"}
Oct 30 20:32:04.024696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.024638    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg" event={"ID":"ceb3f0f5-e299-4472-aa62-6936c4cf8bbc","Type":"ContainerDied","Data":"c335e3dd67e189f52652ff397d6ff4fe56530a63567cda87bd6070d5d1676f85"}
Oct 30 20:32:04.035305 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.035285    4519 scope.go:122] "RemoveContainer" containerID="f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69"
Oct 30 20:32:04.044669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.044643    4519 scope.go:122] "RemoveContainer" containerID="2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180"
Oct 30 20:32:04.044952 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:32:04.044933    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180\": container with ID starting with 2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180 not found: ID does not exist" containerID="2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180"
Oct 30 20:32:04.045026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.044964    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180"} err="failed to get container status \"2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180\": rpc error: code = NotFound desc = could not find container \"2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180\": container with ID starting with 2c3593b91b0b5e72cc93412bf2b441c3c3ef0661f764c818a4a51058b5359180 not found: ID does not exist"
Oct 30 20:32:04.045026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.044984    4519 scope.go:122] "RemoveContainer" containerID="f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69"
Oct 30 20:32:04.045236 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:32:04.045221    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69\": container with ID starting with f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69 not found: ID does not exist" containerID="f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69"
Oct 30 20:32:04.045289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.045239    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69"} err="failed to get container status \"f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69\": rpc error: code = NotFound desc = could not find container \"f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69\": container with ID starting with f677a0d3cb280b3305b46d1d3d446d9806f56c3b30dea20c14fd99e018c23e69 not found: ID does not exist"
Oct 30 20:32:04.045289 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.045253    4519 scope.go:122] "RemoveContainer" containerID="4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2"
Oct 30 20:32:04.056871 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.056855    4519 scope.go:122] "RemoveContainer" containerID="4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2"
Oct 30 20:32:04.057149 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:32:04.057120    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2\": container with ID starting with 4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2 not found: ID does not exist" containerID="4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2"
Oct 30 20:32:04.057277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.057157    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2"} err="failed to get container status \"4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2\": rpc error: code = NotFound desc = could not find container \"4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2\": container with ID starting with 4108d44c5e24af8bb8e6b3569b768d819d6136a9088749ba3af4a4d8a635d2d2 not found: ID does not exist"
Oct 30 20:32:04.079250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.079208    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-xbjnv"]
Oct 30 20:32:04.094705 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.094672    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-xbjnv"]
Oct 30 20:32:04.129181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.129141    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"]
Oct 30 20:32:04.135702 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:04.135669    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-lckvg"]
Oct 30 20:32:05.096626 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.096550    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f86dcf78c-pz9gr"]
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097035    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097055    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097080    4519 cpu_manager.go:414] "containerMap: removing container" podUID="dae99e72-5490-4883-be29-e73f0f85d920" containerName="payload-pre-processing"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097087    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae99e72-5490-4883-be29-e73f0f85d920" containerName="payload-pre-processing"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097115    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerName="manager"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097121    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerName="manager"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097139    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cbe26dba-8ed3-4ea9-8f62-cf6829667465" containerName="revoke-keys"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097146    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe26dba-8ed3-4ea9-8f62-cf6829667465" containerName="revoke-keys"
Oct 30 20:32:05.097154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097159    4519 cpu_manager.go:414] "containerMap: removing container" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097168    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097186    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerName="manager"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097193    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerName="manager"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097203    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097209    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097289    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6881d466-0536-4777-8072-0dc9d3733166" containerName="istio-proxy"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097300    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerName="manager"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097309    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe26dba-8ed3-4ea9-8f62-cf6829667465" containerName="revoke-keys"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097319    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ff036e-01cc-42be-8059-e4721ce56fbc" containerName="main"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097329    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" containerName="manager"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097338    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" containerName="istio-proxy"
Oct 30 20:32:05.097682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.097348    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae99e72-5490-4883-be29-e73f0f85d920" containerName="payload-pre-processing"
Oct 30 20:32:05.102561 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.102536    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.131857 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.131831    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:32:05.132045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.131946    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:32:05.132045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.131988    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:32:05.132045 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.132034    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nw5kj\""
Oct 30 20:32:05.135082 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.135056    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-pz9gr"]
Oct 30 20:32:05.140860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.140835    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-webhook-cert\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.140933 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.140896    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgxw\" (UniqueName: \"kubernetes.io/projected/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-kube-api-access-pqgxw\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.141047 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.141022    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-metrics-certs\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.241616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.241567    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-webhook-cert\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.241616 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.241631    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgxw\" (UniqueName: \"kubernetes.io/projected/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-kube-api-access-pqgxw\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.241846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.241671    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-metrics-certs\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.244162 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.244132    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-metrics-certs\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.244291 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.244174    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-webhook-cert\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.260569 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.260541    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgxw\" (UniqueName: \"kubernetes.io/projected/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-kube-api-access-pqgxw\") pod \"maas-controller-f86dcf78c-pz9gr\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") " pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.416261 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.416223    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:05.610860 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.610829    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9324bb-be21-4f9f-a5cd-a983a2d585aa" path="/var/lib/kubelet/pods/ae9324bb-be21-4f9f-a5cd-a983a2d585aa/volumes"
Oct 30 20:32:05.611276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.611258    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb3f0f5-e299-4472-aa62-6936c4cf8bbc" path="/var/lib/kubelet/pods/ceb3f0f5-e299-4472-aa62-6936c4cf8bbc/volumes"
Oct 30 20:32:05.729126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.729099    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-pz9gr"]
Oct 30 20:32:05.729909 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:32:05.729882    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c729fd4_ac56_44fb_b3a6_ee64c9a07bf7.slice/crio-8951f0523fa59f43367508ba0073cecf8dcc976007d8b8127aa26c5f8cc64eaa WatchSource:0}: Error finding container 8951f0523fa59f43367508ba0073cecf8dcc976007d8b8127aa26c5f8cc64eaa: Status 404 returned error can't find the container with id 8951f0523fa59f43367508ba0073cecf8dcc976007d8b8127aa26c5f8cc64eaa
Oct 30 20:32:05.731144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:05.731119    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:32:06.043577 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:06.043489    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-pz9gr" event={"ID":"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7","Type":"ContainerStarted","Data":"8951f0523fa59f43367508ba0073cecf8dcc976007d8b8127aa26c5f8cc64eaa"}
Oct 30 20:32:07.062275 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:07.062222    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-pz9gr" event={"ID":"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7","Type":"ContainerStarted","Data":"00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec"}
Oct 30 20:32:07.062839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:07.062355    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:07.105259 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:07.105196    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f86dcf78c-pz9gr" podStartSLOduration=1.648509871 podStartE2EDuration="2.105180371s" totalImagesPullingTime="456.6705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:07.102379198 +0000 UTC m=+3451.603082922" watchObservedRunningTime="2026-10-30 20:32:07.105180371 +0000 UTC m=+3451.605884094"
Oct 30 20:32:18.071450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:18.071417    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:32.219895 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:32:32.219816    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:32:36.643196 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:36.643150    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-pz9gr"]
Oct 30 20:32:36.643722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:36.643368    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-f86dcf78c-pz9gr" podUID="0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" containerName="manager" containerID="cri-o://00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec" gracePeriod=30
Oct 30 20:32:36.890969 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:36.890933    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:37.029886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.029852    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-webhook-cert\") pod \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") "
Oct 30 20:32:37.030068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.029968    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-kube-api-access-pqgxw\" (UniqueName: \"kubernetes.io/projected/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-kube-api-access-pqgxw\") pod \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") "
Oct 30 20:32:37.030068 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.030018    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-metrics-certs\") pod \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\" (UID: \"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7\") "
Oct 30 20:32:37.032054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.032019    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-metrics-certs" pod "0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" (UID: "0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:37.032054 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.032035    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-webhook-cert" pod "0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" (UID: "0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:37.032270 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.032082    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-kube-api-access-pqgxw" pod "0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" (UID: "0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7"). InnerVolumeSpecName "kube-api-access-pqgxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:37.131098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.131058    4519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-metrics-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:37.131098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.131092    4519 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-webhook-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:37.131098 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.131101    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqgxw\" (UniqueName: \"kubernetes.io/projected/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7-kube-api-access-pqgxw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:32:37.265899 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.265864    4519 generic.go:356] "Generic (PLEG): container finished" podID="0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" containerID="00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec" exitCode=0
Oct 30 20:32:37.266107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.265910    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-pz9gr"
Oct 30 20:32:37.266107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.265934    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-pz9gr" event={"ID":"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7","Type":"ContainerDied","Data":"00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec"}
Oct 30 20:32:37.266107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.265987    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-pz9gr" event={"ID":"0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7","Type":"ContainerDied","Data":"8951f0523fa59f43367508ba0073cecf8dcc976007d8b8127aa26c5f8cc64eaa"}
Oct 30 20:32:37.266107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.266010    4519 scope.go:122] "RemoveContainer" containerID="00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec"
Oct 30 20:32:37.276354 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.276339    4519 scope.go:122] "RemoveContainer" containerID="00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec"
Oct 30 20:32:37.276675 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:32:37.276654    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec\": container with ID starting with 00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec not found: ID does not exist" containerID="00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec"
Oct 30 20:32:37.276763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.276682    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec"} err="failed to get container status \"00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec\": rpc error: code = NotFound desc = could not find container \"00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec\": container with ID starting with 00368e6e47337309e33020eb614b17cff6ad61273960ca41913d1757b66ab8ec not found: ID does not exist"
Oct 30 20:32:37.317586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.317507    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-pz9gr"]
Oct 30 20:32:37.329631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.329605    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-pz9gr"]
Oct 30 20:32:37.611274 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:37.611190    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" path="/var/lib/kubelet/pods/0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7/volumes"
Oct 30 20:32:38.526333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.526297    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f86dcf78c-2cssg"]
Oct 30 20:32:38.526791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.526710    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" containerName="manager"
Oct 30 20:32:38.526791 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.526722    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" containerName="manager"
Oct 30 20:32:38.526880 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.526797    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c729fd4-ac56-44fb-b3a6-ee64c9a07bf7" containerName="manager"
Oct 30 20:32:38.531513 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.531491    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.546290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.546265    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:32:38.558394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.558353    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:32:38.558394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.558353    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nw5kj\""
Oct 30 20:32:38.558606 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.558534    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:32:38.561858 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.561835    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-2cssg"]
Oct 30 20:32:38.642604 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.642568    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c18634-113b-4c35-aef2-29d15f9b545a-metrics-certs\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.642781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.642618    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5b6d\" (UniqueName: \"kubernetes.io/projected/54c18634-113b-4c35-aef2-29d15f9b545a-kube-api-access-d5b6d\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.642781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.642697    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54c18634-113b-4c35-aef2-29d15f9b545a-webhook-cert\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.743563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.743528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c18634-113b-4c35-aef2-29d15f9b545a-metrics-certs\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.743745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.743586    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5b6d\" (UniqueName: \"kubernetes.io/projected/54c18634-113b-4c35-aef2-29d15f9b545a-kube-api-access-d5b6d\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.743745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.743631    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54c18634-113b-4c35-aef2-29d15f9b545a-webhook-cert\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.746179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.746154    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54c18634-113b-4c35-aef2-29d15f9b545a-webhook-cert\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.746283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.746212    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c18634-113b-4c35-aef2-29d15f9b545a-metrics-certs\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.762824 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.762795    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5b6d\" (UniqueName: \"kubernetes.io/projected/54c18634-113b-4c35-aef2-29d15f9b545a-kube-api-access-d5b6d\") pod \"maas-controller-f86dcf78c-2cssg\" (UID: \"54c18634-113b-4c35-aef2-29d15f9b545a\") " pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:38.844718 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:38.844646    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:39.148010 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:39.147854    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f86dcf78c-2cssg"]
Oct 30 20:32:39.150795 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:32:39.150759    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c18634_113b_4c35_aef2_29d15f9b545a.slice/crio-dab3a2c9fca0a5a64f82376a28269dc94911e65c1fce605d751bf896e8d03113 WatchSource:0}: Error finding container dab3a2c9fca0a5a64f82376a28269dc94911e65c1fce605d751bf896e8d03113: Status 404 returned error can't find the container with id dab3a2c9fca0a5a64f82376a28269dc94911e65c1fce605d751bf896e8d03113
Oct 30 20:32:39.281613 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:39.281571    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-2cssg" event={"ID":"54c18634-113b-4c35-aef2-29d15f9b545a","Type":"ContainerStarted","Data":"dab3a2c9fca0a5a64f82376a28269dc94911e65c1fce605d751bf896e8d03113"}
Oct 30 20:32:40.289719 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:40.289677    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f86dcf78c-2cssg" event={"ID":"54c18634-113b-4c35-aef2-29d15f9b545a","Type":"ContainerStarted","Data":"6344d30de64241bfc52e4c22e42e446aa13f5e5aa02744118e69a4e35c26c72a"}
Oct 30 20:32:40.290140 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:40.289751    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:32:40.331423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:40.331372    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f86dcf78c-2cssg" podStartSLOduration=2.070448867 podStartE2EDuration="2.331357708s" totalImagesPullingTime="260.908841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:40.329961682 +0000 UTC m=+3484.830665406" watchObservedRunningTime="2026-10-30 20:32:40.331357708 +0000 UTC m=+3484.832061431"
Oct 30 20:32:45.806557 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:45.806517    4519 scope.go:122] "RemoveContainer" containerID="6f23f906d073cb743b00da8ec176a7dac8758cf46581beb90c5b1861aae7a5eb"
Oct 30 20:32:45.817083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:45.817062    4519 scope.go:122] "RemoveContainer" containerID="f2e435b025bafbc9508bedc0573f7ac76af1bbd8e334cccad5932f4e23cc3aba"
Oct 30 20:32:45.826120 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:45.826099    4519 scope.go:122] "RemoveContainer" containerID="b2da3b0de192e76a7f24e997580e156292c9d15e7284cedae20e4e5f5b3279ef"
Oct 30 20:32:45.834701 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:45.834676    4519 scope.go:122] "RemoveContainer" containerID="38fe93ddc8632d04f19b9c6ab54ccc0d6e67d0fb50de28ff2cce0263357082d5"
Oct 30 20:32:45.843066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:45.843048    4519 scope.go:122] "RemoveContainer" containerID="c703d1173df035a8e8715780c7310d550cea4ddaa1740d142eef58d3b349dcf5"
Oct 30 20:32:45.851421 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:45.851403    4519 scope.go:122] "RemoveContainer" containerID="70da87e188afd99be5c7fed7744169b00de9ca416c2a71c5310d4b690c43bda6"
Oct 30 20:32:45.859840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:45.859818    4519 scope.go:122] "RemoveContainer" containerID="d08a5b186cb0e67fec19d748b7e95a3c1eaaaae90671eab535b9a744be3f8ada"
Oct 30 20:32:51.298825 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:32:51.298786    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f86dcf78c-2cssg"
Oct 30 20:33:02.219137 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:33:02.219095    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:33:32.219964 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:33:32.219919    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:33:45.664004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.663968    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"]
Oct 30 20:33:45.667625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.667597    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.695044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.694998    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"]
Oct 30 20:33:45.698625 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.698590    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"]
Oct 30 20:33:45.702180 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.702159    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.725072 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.725041    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"]
Oct 30 20:33:45.737572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737539    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.737720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737578    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.737720 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737665    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.737815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737712    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.737815 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737799    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.737897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737830    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.737897 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737854    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.737977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737903    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.737977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737937    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4nr\" (UniqueName: \"kubernetes.io/projected/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-kube-api-access-cj4nr\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.738052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.737975    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb28c\" (UniqueName: \"kubernetes.io/projected/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-kube-api-access-cb28c\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.738052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738014    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.738052 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738033    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.738164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738060    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.738164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738112    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.738238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738188    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.738238 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738220    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.738364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738293    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.738364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.738310    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.838822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838783    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.838822 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838842    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838859    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838887    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838908    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838929    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839091 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.838960    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839323    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839347    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839403    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839403    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839451 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839413    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839481    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4nr\" (UniqueName: \"kubernetes.io/projected/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-kube-api-access-cj4nr\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839513    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb28c\" (UniqueName: \"kubernetes.io/projected/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-kube-api-access-cb28c\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839550    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839577    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839645    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839692    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839709    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.839834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839811    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.840298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839898    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.840298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839915    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.840298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839929    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.840298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.839962    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.840566 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.840300    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.840635 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.840558    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.841522 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.841487    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.841643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.841545    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.842109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.842083    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.842284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.842264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.859154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.859131    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.859298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.859203    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.859399 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.859382    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4nr\" (UniqueName: \"kubernetes.io/projected/0cf4fb75-5c06-41d1-904b-a1b6f0784eb4-kube-api-access-cj4nr\") pod \"maas-default-gateway-openshift-default-75bf867b6c-88mz8\" (UID: \"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:45.859508 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.859490    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb28c\" (UniqueName: \"kubernetes.io/projected/24bb6760-98c6-4839-bfe7-6a6a5d42dca6-kube-api-access-cb28c\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8zcb4\" (UID: \"24bb6760-98c6-4839-bfe7-6a6a5d42dca6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:45.927931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.927862    4519 scope.go:122] "RemoveContainer" containerID="badb0ae4c693b2ee08ab8611fd91d97ba50213b1c8a1df74f014cbeb38b29fba"
Oct 30 20:33:45.980297 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:45.980268    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:46.017582 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.017533    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:46.301916 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.301879    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"]
Oct 30 20:33:46.302367 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:33:46.302329    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bb6760_98c6_4839_bfe7_6a6a5d42dca6.slice/crio-6756b3736ef261f20868e784c8251855a7855d975cc0a25a1d7b43005e158dde WatchSource:0}: Error finding container 6756b3736ef261f20868e784c8251855a7855d975cc0a25a1d7b43005e158dde: Status 404 returned error can't find the container with id 6756b3736ef261f20868e784c8251855a7855d975cc0a25a1d7b43005e158dde
Oct 30 20:33:46.304636 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.304605    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:46.304712 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.304668    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:46.304712 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.304698    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:46.343107 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:33:46.343078    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf4fb75_5c06_41d1_904b_a1b6f0784eb4.slice/crio-91f490373c26be7e639e6ef25a7d55cadeae028089f396f57d5f09bc608a4984 WatchSource:0}: Error finding container 91f490373c26be7e639e6ef25a7d55cadeae028089f396f57d5f09bc608a4984: Status 404 returned error can't find the container with id 91f490373c26be7e639e6ef25a7d55cadeae028089f396f57d5f09bc608a4984
Oct 30 20:33:46.343277 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.343248    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"]
Oct 30 20:33:46.345610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.345579    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:46.345725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.345654    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:46.345725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.345701    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:46.729911 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.729863    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" event={"ID":"24bb6760-98c6-4839-bfe7-6a6a5d42dca6","Type":"ContainerStarted","Data":"c0aedb43d2b4cda2a8ba4a26d01c019b492a70bce38182476d40bbd3d21b43aa"}
Oct 30 20:33:46.730371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.729941    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" event={"ID":"24bb6760-98c6-4839-bfe7-6a6a5d42dca6","Type":"ContainerStarted","Data":"6756b3736ef261f20868e784c8251855a7855d975cc0a25a1d7b43005e158dde"}
Oct 30 20:33:46.732966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.732877    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" event={"ID":"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4","Type":"ContainerStarted","Data":"58ec82fc0c7d0475bb402b159113dd9f340545dab60fc4351c309a1e514c409c"}
Oct 30 20:33:46.732966 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.732927    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" event={"ID":"0cf4fb75-5c06-41d1-904b-a1b6f0784eb4","Type":"ContainerStarted","Data":"91f490373c26be7e639e6ef25a7d55cadeae028089f396f57d5f09bc608a4984"}
Oct 30 20:33:46.843951 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.843887    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" podStartSLOduration=1.843867449 podStartE2EDuration="1.843867449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:46.798067056 +0000 UTC m=+3551.298770791" watchObservedRunningTime="2026-10-30 20:33:46.843867449 +0000 UTC m=+3551.344571169"
Oct 30 20:33:46.844143 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.844017    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podStartSLOduration=1.844011944 podStartE2EDuration="1.844011944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:46.841190496 +0000 UTC m=+3551.341894221" watchObservedRunningTime="2026-10-30 20:33:46.844011944 +0000 UTC m=+3551.344715658"
Oct 30 20:33:46.982555 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:46.981559    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:47.018217 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:47.018174    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:47.019350 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:47.019325    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-88mz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body=
Oct 30 20:33:47.019427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:47.019372    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podUID="0cf4fb75-5c06-41d1-904b-a1b6f0784eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused"
Oct 30 20:33:47.983492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:47.982591    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8zcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Oct 30 20:33:47.983492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:47.982661    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" podUID="24bb6760-98c6-4839-bfe7-6a6a5d42dca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Oct 30 20:33:48.018765 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:48.018727    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-88mz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body=
Oct 30 20:33:48.018950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:48.018788    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podUID="0cf4fb75-5c06-41d1-904b-a1b6f0784eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused"
Oct 30 20:33:48.981041 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:48.980939    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8zcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Oct 30 20:33:48.981938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:48.981447    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" podUID="24bb6760-98c6-4839-bfe7-6a6a5d42dca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Oct 30 20:33:49.018253 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:49.018216    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-88mz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body=
Oct 30 20:33:49.018768 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:49.018738    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podUID="0cf4fb75-5c06-41d1-904b-a1b6f0784eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused"
Oct 30 20:33:49.984004 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:49.983959    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8zcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Oct 30 20:33:49.984347 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:49.984310    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" podUID="24bb6760-98c6-4839-bfe7-6a6a5d42dca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Oct 30 20:33:50.018453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:50.018410    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-88mz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body=
Oct 30 20:33:50.018902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:50.018488    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podUID="0cf4fb75-5c06-41d1-904b-a1b6f0784eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused"
Oct 30 20:33:50.981995 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:50.981946    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8zcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Oct 30 20:33:50.982442 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:50.982403    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" podUID="24bb6760-98c6-4839-bfe7-6a6a5d42dca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Oct 30 20:33:51.017963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:51.017926    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-88mz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body=
Oct 30 20:33:51.018178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:51.018123    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podUID="0cf4fb75-5c06-41d1-904b-a1b6f0784eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused"
Oct 30 20:33:51.980971 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:51.980925    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8zcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Oct 30 20:33:51.981641 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:51.981599    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" podUID="24bb6760-98c6-4839-bfe7-6a6a5d42dca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Oct 30 20:33:52.017885 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:52.017852    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-88mz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body=
Oct 30 20:33:52.018117 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:52.018098    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podUID="0cf4fb75-5c06-41d1-904b-a1b6f0784eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused"
Oct 30 20:33:52.981861 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:52.981811    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8zcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused" start-of-body=
Oct 30 20:33:52.982321 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:52.981889    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4" podUID="24bb6760-98c6-4839-bfe7-6a6a5d42dca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.103:15021/healthz/ready\": dial tcp 10.133.1.103:15021: connect: connection refused"
Oct 30 20:33:53.018170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:53.018132    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-88mz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body=
Oct 30 20:33:53.018353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:53.018202    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8" podUID="0cf4fb75-5c06-41d1-904b-a1b6f0784eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused"
Oct 30 20:33:54.708168 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:54.708127    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:54.709154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:54.709128    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:54.709943 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:54.709920    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8zcb4"
Oct 30 20:33:54.797298 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:54.797264    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:54.797917 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:54.797882    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:33:54.798688 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:33:54.798665    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-88mz8"
Oct 30 20:34:02.234247 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:34:02.234194    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:34:32.220055 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:34:32.220015    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:34:45.951243 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:45.951207    4519 scope.go:122] "RemoveContainer" containerID="e868961abf9817e22fa2ce96a1c948a8a1c7d8f12cb9f8679fa35d974533b03c"
Oct 30 20:34:45.962435 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:45.962411    4519 scope.go:122] "RemoveContainer" containerID="c8e77b89a2546f0fde83a27b17dc909dbfa495db702d3ccf862b28bf5323707c"
Oct 30 20:34:51.033278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:51.033222    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"]
Oct 30 20:34:51.043265 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:51.043227    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"]
Oct 30 20:34:51.046012 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:51.045983    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdc971-8g2mp"]
Oct 30 20:34:51.058488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:51.056634    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a631379-scgc2"]
Oct 30 20:34:51.611698 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:51.611659    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e" path="/var/lib/kubelet/pods/ef6b985c-5bd9-4bff-8c0b-b4921fbecd8e/volumes"
Oct 30 20:34:51.612122 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:34:51.612107    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5432e4-3ab4-4f60-95a2-0322a5326808" path="/var/lib/kubelet/pods/fe5432e4-3ab4-4f60-95a2-0322a5326808/volumes"
Oct 30 20:35:01.028925 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:01.028884    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"]
Oct 30 20:35:01.040691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:01.040652    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aab56ab-b8h7m"]
Oct 30 20:35:01.611187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:01.611139    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e58824-5528-42b1-9d5c-3852f6876cd5" path="/var/lib/kubelet/pods/02e58824-5528-42b1-9d5c-3852f6876cd5/volumes"
Oct 30 20:35:02.027775 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:02.027734    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"]
Oct 30 20:35:02.039630 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:02.039594    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a93742-lrffp"]
Oct 30 20:35:02.220146 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:35:02.220112    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:35:03.610664 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:03.610629    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5024961-e814-4fb2-953a-38f1920d7a08" path="/var/lib/kubelet/pods/a5024961-e814-4fb2-953a-38f1920d7a08/volumes"
Oct 30 20:35:14.029884 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:14.029842    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"]
Oct 30 20:35:14.042192 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:14.042150    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"]
Oct 30 20:35:14.044740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:14.044709    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ec88d4-5w74p"]
Oct 30 20:35:14.046708 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:14.046688    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1301367-w77z6"]
Oct 30 20:35:15.611319 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:15.611284    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d" path="/var/lib/kubelet/pods/617c0e40-57e3-4c02-b4f3-5a7c2ac7c55d/volumes"
Oct 30 20:35:15.611773 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:15.611757    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cded2d35-8c43-4b09-aa44-24b0bc2600d5" path="/var/lib/kubelet/pods/cded2d35-8c43-4b09-aa44-24b0bc2600d5/volumes"
Oct 30 20:35:16.031833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:16.031790    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"]
Oct 30 20:35:16.041422 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:16.041393    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad220342-7hrzf"]
Oct 30 20:35:17.611271 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:17.611236    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480db7f0-53bf-4af2-8ad2-f7d92fc138ab" path="/var/lib/kubelet/pods/480db7f0-53bf-4af2-8ad2-f7d92fc138ab/volumes"
Oct 30 20:35:32.301491 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:35:32.283643    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:35:45.998343 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:45.998305    4519 scope.go:122] "RemoveContainer" containerID="8d89fffc778a034c1dc8d6c4b627e098bc07aa4aff183b92cfd5265822caeab2"
Oct 30 20:35:46.007945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:46.007920    4519 scope.go:122] "RemoveContainer" containerID="3f3228b54c505967aa9a9165862e47dfad40d2a8bd800286c7903d2e2779d4b2"
Oct 30 20:35:46.016633 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:46.016612    4519 scope.go:122] "RemoveContainer" containerID="370920a3c90651a861f58cc9999f52a3feb5b51674e804d61047e3a3730c977d"
Oct 30 20:35:46.024763 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:46.024739    4519 scope.go:122] "RemoveContainer" containerID="0657595e4ccd522b20998a96c10b420c96c175928e8e355d2ed72a6ebd6ec0a6"
Oct 30 20:35:46.032728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:35:46.032699    4519 scope.go:122] "RemoveContainer" containerID="2eb22be62f5442deaf7039e2428d12c404fdd10a2429454b3c779c3821fd5d09"
Oct 30 20:36:02.219573 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:36:02.219520    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:36:25.040075 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:25.040039    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"]
Oct 30 20:36:25.052126 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:25.052083    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"]
Oct 30 20:36:25.054290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:25.054261    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09fec3d-m2lcb"]
Oct 30 20:36:25.056170 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:25.056148    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67617f59-9wzph"]
Oct 30 20:36:25.611439 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:25.611398    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4633c0f2-30f0-48e9-b494-9b7d89323df2" path="/var/lib/kubelet/pods/4633c0f2-30f0-48e9-b494-9b7d89323df2/volumes"
Oct 30 20:36:25.611950 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:25.611926    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a147ff-5aca-4876-952a-6c05cb21b6c4" path="/var/lib/kubelet/pods/a8a147ff-5aca-4876-952a-6c05cb21b6c4/volumes"
Oct 30 20:36:29.030818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:29.030781    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"]
Oct 30 20:36:29.043764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:29.043733    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3b23c0-8hprg"]
Oct 30 20:36:29.045740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:29.045728    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"]
Oct 30 20:36:29.047538 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:29.047515    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e5d57-82lc9"]
Oct 30 20:36:29.610846 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:29.610814    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3994718d-da61-4ea1-9bed-2daa3939642a" path="/var/lib/kubelet/pods/3994718d-da61-4ea1-9bed-2daa3939642a/volumes"
Oct 30 20:36:29.611250 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:29.611236    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d48e57-30fa-43a6-8d31-cd1a4fec6b47" path="/var/lib/kubelet/pods/c7d48e57-30fa-43a6-8d31-cd1a4fec6b47/volumes"
Oct 30 20:36:32.219860 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:36:32.219819    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:36:39.031223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:39.031177    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"]
Oct 30 20:36:39.043210 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:39.043175    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30a5110-btmm9"]
Oct 30 20:36:39.611333 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:39.611298    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe26dba-8ed3-4ea9-8f62-cf6829667465" path="/var/lib/kubelet/pods/cbe26dba-8ed3-4ea9-8f62-cf6829667465/volumes"
Oct 30 20:36:46.076278 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.076239    4519 scope.go:122] "RemoveContainer" containerID="087e170fc8b91c8da1bbce03497c6732685e549305bfeed1f216fa20a7c7a49c"
Oct 30 20:36:46.086982 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.086961    4519 scope.go:122] "RemoveContainer" containerID="3e573943e2ba8878e77715ff218b4eb226fd9e05cff5467662bed3283fad4de8"
Oct 30 20:36:46.096108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.096083    4519 scope.go:122] "RemoveContainer" containerID="d23bbbaed24befa91efeb119dba8dd7115ed612e462ced25e864b88c12c077e6"
Oct 30 20:36:46.104397 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.104372    4519 scope.go:122] "RemoveContainer" containerID="e0c96eccfbb2d4623c30a2dd803b961f1d949418e747f67b8b9f37e4081cc047"
Oct 30 20:36:46.112770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.112745    4519 scope.go:122] "RemoveContainer" containerID="f85526df8905da4cd589e888aedc9947f9a3ec1e372805b80b0f9d8ae564a070"
Oct 30 20:36:46.120839 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.120816    4519 scope.go:122] "RemoveContainer" containerID="d0d5d19408c501c99d76e218847312a8736bcf80e67a41913a715b1f4a11356f"
Oct 30 20:36:46.129428 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.129397    4519 scope.go:122] "RemoveContainer" containerID="a668e14e8e321d66d87055881fa024d406ca73d2a44692d058e31406fa5c5286"
Oct 30 20:36:46.138158 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.138138    4519 scope.go:122] "RemoveContainer" containerID="b8ae50d62e3bbf57a954ba5774cfd35c53833b24bac8bd48ecb7fcc7986b8b4c"
Oct 30 20:36:46.146022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.146001    4519 scope.go:122] "RemoveContainer" containerID="d7f0f306aec4737c6b0e997142bddd35e6ab95e52de712c70e5a9c491af0dc53"
Oct 30 20:36:46.155766 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.155744    4519 scope.go:122] "RemoveContainer" containerID="84c9b5ee07334f81e9b0d057a1569df1110bfba590f5abd35e7aa34494242e5f"
Oct 30 20:36:46.163963 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.163940    4519 scope.go:122] "RemoveContainer" containerID="18fbd58e91af3c35036cb8b43e9f7ce868cdf858b5e781cff1ef4aab2f67fa5b"
Oct 30 20:36:46.173548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:36:46.173526    4519 scope.go:122] "RemoveContainer" containerID="2aa5233c6cf5cb2f2fd62b3f641dacae49bc35bdedba4f654510616546ee0897"
Oct 30 20:37:02.220351 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:37:02.220305    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:37:32.220811 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:37:32.220768    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:37:46.262489 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.259635    4519 scope.go:122] "RemoveContainer" containerID="e612f6b268829fb7d1ecc42e4d38699ed4b6f69ead7ac6178c7a838a807e0517"
Oct 30 20:37:46.310179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.310147    4519 scope.go:122] "RemoveContainer" containerID="a30af797c1f780a9ba34db8dec79c132354ced33bc3c28834ae90e8c8cddeecd"
Oct 30 20:37:46.336407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.336380    4519 scope.go:122] "RemoveContainer" containerID="94bf131ebed7e1c2cb43846c906068489a6b47c00c5e46b14a0840d90f5d9273"
Oct 30 20:37:46.345527 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.345483    4519 scope.go:122] "RemoveContainer" containerID="87c827180231d02eedefdb8531a482e8e16c498339ad8d3504956ee3373afec3"
Oct 30 20:37:46.382740 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.382611    4519 scope.go:122] "RemoveContainer" containerID="710499f8eba5d4a85788a541f60e6b3f2432fd747b4a695cc860d40c844bb865"
Oct 30 20:37:46.426139 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.426024    4519 scope.go:122] "RemoveContainer" containerID="0f484e22994b9595d72c6fd0c4115bf6d2db5aca16e9b7cd89769dfb22cebb34"
Oct 30 20:37:46.439266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.439191    4519 scope.go:122] "RemoveContainer" containerID="3fe1f390ae757fc4f6bf9e9afdda2eebfcbd25fdd2dc27d9624719e82c7dc485"
Oct 30 20:37:46.505956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.505825    4519 scope.go:122] "RemoveContainer" containerID="15f0311e047aac9a2350e13a4ed50a8812ae1d918f07ec05e22799a89518a82e"
Oct 30 20:37:46.532716 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.532689    4519 scope.go:122] "RemoveContainer" containerID="35411345bbfd2e77f5382335fb61350997cd020b9ffbe2cde4fb5aab15b43718"
Oct 30 20:37:46.543003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:37:46.542980    4519 scope.go:122] "RemoveContainer" containerID="295b898777ac6c261e23ebca58603ba69fa1c92d9de300be16d1c01217041b64"
Oct 30 20:38:02.220196 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:38:02.220155    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:38:15.829572 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:15.829524    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"]
Oct 30 20:38:15.833044 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:15.833018    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:15.848672 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:15.848642    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-26e5b0\"/\"openshift-service-ca.crt\""
Oct 30 20:38:15.859584 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:15.859549    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-26e5b0\"/\"kube-root-ca.crt\""
Oct 30 20:38:15.859756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:15.859605    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-26e5b0\"/\"default-dockercfg-zjbmz\""
Oct 30 20:38:15.859991 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:15.859976    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-26e5b0\"/\"simulator-single-main-26e5b0-kserve-self-signed-certs\""
Oct 30 20:38:15.863064 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:15.863042    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"]
Oct 30 20:38:16.020402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.020364    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-model-cache\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.020603 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.020410    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-dshm\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.020603 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.020512    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg24b\" (UniqueName: \"kubernetes.io/projected/4088c15c-949b-4a90-a062-d5e341fcd06c-kube-api-access-sg24b\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.020603 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.020597    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4088c15c-949b-4a90-a062-d5e341fcd06c-tls-certs\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.020728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.020632    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-tmp-dir\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.020728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.020649    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-home\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.121779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.121675    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-model-cache\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.121779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.121738    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-dshm\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.121779 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.121776    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg24b\" (UniqueName: \"kubernetes.io/projected/4088c15c-949b-4a90-a062-d5e341fcd06c-kube-api-access-sg24b\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.122109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.121833    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4088c15c-949b-4a90-a062-d5e341fcd06c-tls-certs\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.122109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.121878    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-tmp-dir\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.122109 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.121899    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-home\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.122285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.122196    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-model-cache\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.122285 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.122209    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-tmp-dir\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.122391 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.122326    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-home\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.124273 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.124244    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-dshm\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.124459 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.124437    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4088c15c-949b-4a90-a062-d5e341fcd06c-tls-certs\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.141426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.141397    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg24b\" (UniqueName: \"kubernetes.io/projected/4088c15c-949b-4a90-a062-d5e341fcd06c-kube-api-access-sg24b\") pod \"simulator-single-main-26e5b0-kserve-5c987b777d-22dck\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") " pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.161295 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.161263    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:16.466125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.466093    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"]
Oct 30 20:38:16.468299 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:38:16.468263    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4088c15c_949b_4a90_a062_d5e341fcd06c.slice/crio-233f8e934362285cef2f07334e7f10116a0c4aa0a392409c2d125d7ef7925547 WatchSource:0}: Error finding container 233f8e934362285cef2f07334e7f10116a0c4aa0a392409c2d125d7ef7925547: Status 404 returned error can't find the container with id 233f8e934362285cef2f07334e7f10116a0c4aa0a392409c2d125d7ef7925547
Oct 30 20:38:16.704939 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.704889    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck" event={"ID":"4088c15c-949b-4a90-a062-d5e341fcd06c","Type":"ContainerStarted","Data":"5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b"}
Oct 30 20:38:16.705150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.704951    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck" event={"ID":"4088c15c-949b-4a90-a062-d5e341fcd06c","Type":"ContainerStarted","Data":"233f8e934362285cef2f07334e7f10116a0c4aa0a392409c2d125d7ef7925547"}
Oct 30 20:38:16.753101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:16.752985    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck" podStartSLOduration=1.75296759 podStartE2EDuration="1.75296759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:16.748768361 +0000 UTC m=+3821.249472082" watchObservedRunningTime="2026-10-30 20:38:16.75296759 +0000 UTC m=+3821.253671316"
Oct 30 20:38:17.161715 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:17.161626    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:26.162144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:26.162048    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:26.178191 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:26.178154    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:26.767787 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:38:26.767755    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:38:32.220579 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:38:32.220451    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:39:01.530144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:01.530100    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"]
Oct 30 20:39:02.221551 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:39:02.221510    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:39:16.551384 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:16.551260    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck" podUID="4088c15c-949b-4a90-a062-d5e341fcd06c" containerName="main" containerID="cri-o://5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b" gracePeriod=45
Oct 30 20:39:16.888894 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:16.888866    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:39:17.004924 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.004887    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4088c15c-949b-4a90-a062-d5e341fcd06c-kube-api-access-sg24b\" (UniqueName: \"kubernetes.io/projected/4088c15c-949b-4a90-a062-d5e341fcd06c-kube-api-access-sg24b\") pod \"4088c15c-949b-4a90-a062-d5e341fcd06c\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") "
Oct 30 20:39:17.005107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.004959    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-tmp-dir\") pod \"4088c15c-949b-4a90-a062-d5e341fcd06c\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") "
Oct 30 20:39:17.005107 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.004978    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-dshm\") pod \"4088c15c-949b-4a90-a062-d5e341fcd06c\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") "
Oct 30 20:39:17.005205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005094    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4088c15c-949b-4a90-a062-d5e341fcd06c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4088c15c-949b-4a90-a062-d5e341fcd06c-tls-certs\") pod \"4088c15c-949b-4a90-a062-d5e341fcd06c\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") "
Oct 30 20:39:17.005205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005147    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-home\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-home\") pod \"4088c15c-949b-4a90-a062-d5e341fcd06c\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") "
Oct 30 20:39:17.005205 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005195    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-model-cache\") pod \"4088c15c-949b-4a90-a062-d5e341fcd06c\" (UID: \"4088c15c-949b-4a90-a062-d5e341fcd06c\") "
Oct 30 20:39:17.005390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005223    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-tmp-dir" pod "4088c15c-949b-4a90-a062-d5e341fcd06c" (UID: "4088c15c-949b-4a90-a062-d5e341fcd06c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:17.005512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005458    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-home" pod "4088c15c-949b-4a90-a062-d5e341fcd06c" (UID: "4088c15c-949b-4a90-a062-d5e341fcd06c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:17.005512 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005503    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-model-cache" pod "4088c15c-949b-4a90-a062-d5e341fcd06c" (UID: "4088c15c-949b-4a90-a062-d5e341fcd06c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:17.005728 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005711    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:17.005795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005733    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:17.005795 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.005748    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:17.007411 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.007384    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4088c15c-949b-4a90-a062-d5e341fcd06c-kube-api-access-sg24b" pod "4088c15c-949b-4a90-a062-d5e341fcd06c" (UID: "4088c15c-949b-4a90-a062-d5e341fcd06c"). InnerVolumeSpecName "kube-api-access-sg24b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:17.007559 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.007532    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-dshm" pod "4088c15c-949b-4a90-a062-d5e341fcd06c" (UID: "4088c15c-949b-4a90-a062-d5e341fcd06c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:17.007692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.007668    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4088c15c-949b-4a90-a062-d5e341fcd06c-tls-certs" pod "4088c15c-949b-4a90-a062-d5e341fcd06c" (UID: "4088c15c-949b-4a90-a062-d5e341fcd06c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:17.026743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.026715    4519 generic.go:356] "Generic (PLEG): container finished" podID="4088c15c-949b-4a90-a062-d5e341fcd06c" containerID="5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b" exitCode=0
Oct 30 20:39:17.026868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.026763    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"
Oct 30 20:39:17.026868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.026766    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck" event={"ID":"4088c15c-949b-4a90-a062-d5e341fcd06c","Type":"ContainerDied","Data":"5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b"}
Oct 30 20:39:17.026868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.026809    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck" event={"ID":"4088c15c-949b-4a90-a062-d5e341fcd06c","Type":"ContainerDied","Data":"233f8e934362285cef2f07334e7f10116a0c4aa0a392409c2d125d7ef7925547"}
Oct 30 20:39:17.026868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.026833    4519 scope.go:122] "RemoveContainer" containerID="5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b"
Oct 30 20:39:17.036495 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.036453    4519 scope.go:122] "RemoveContainer" containerID="5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b"
Oct 30 20:39:17.036767 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:39:17.036745    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b\": container with ID starting with 5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b not found: ID does not exist" containerID="5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b"
Oct 30 20:39:17.036818 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.036777    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b"} err="failed to get container status \"5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b\": rpc error: code = NotFound desc = could not find container \"5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b\": container with ID starting with 5bc2760ad2572b5fa02ad95cda840ac1fc65bb338a7db628e0226611200f0d6b not found: ID does not exist"
Oct 30 20:39:17.081524 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.081495    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"]
Oct 30 20:39:17.085556 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.085532    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck"]
Oct 30 20:39:17.106777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.106743    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg24b\" (UniqueName: \"kubernetes.io/projected/4088c15c-949b-4a90-a062-d5e341fcd06c-kube-api-access-sg24b\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:17.106777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.106771    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4088c15c-949b-4a90-a062-d5e341fcd06c-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:17.106777 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.106783    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4088c15c-949b-4a90-a062-d5e341fcd06c-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:17.611178 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.611144    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4088c15c-949b-4a90-a062-d5e341fcd06c" path="/var/lib/kubelet/pods/4088c15c-949b-4a90-a062-d5e341fcd06c/volumes"
Oct 30 20:39:17.757416 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:17.757370    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-26e5b0/simulator-single-main-26e5b0-kserve-5c987b777d-22dck" podUID="4088c15c-949b-4a90-a062-d5e341fcd06c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:39:32.238102 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:39:32.238053    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:39:42.115420 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.115381    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"]
Oct 30 20:39:42.115902 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.115615    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz" podUID="4669b11d-cc77-431d-8f96-80e296a7e193" containerName="manager" containerID="cri-o://4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901" gracePeriod=10
Oct 30 20:39:42.367912 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.367858    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 20:39:42.469802 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.469764    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4669b11d-cc77-431d-8f96-80e296a7e193-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4669b11d-cc77-431d-8f96-80e296a7e193-extensions-socket-volume\") pod \"4669b11d-cc77-431d-8f96-80e296a7e193\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") "
Oct 30 20:39:42.469972 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.469881    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4669b11d-cc77-431d-8f96-80e296a7e193-kube-api-access-mvslv\" (UniqueName: \"kubernetes.io/projected/4669b11d-cc77-431d-8f96-80e296a7e193-kube-api-access-mvslv\") pod \"4669b11d-cc77-431d-8f96-80e296a7e193\" (UID: \"4669b11d-cc77-431d-8f96-80e296a7e193\") "
Oct 30 20:39:42.470135 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.470111    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4669b11d-cc77-431d-8f96-80e296a7e193-extensions-socket-volume" pod "4669b11d-cc77-431d-8f96-80e296a7e193" (UID: "4669b11d-cc77-431d-8f96-80e296a7e193"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:42.470218 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.470205    4519 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4669b11d-cc77-431d-8f96-80e296a7e193-extensions-socket-volume\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:42.472112 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.472083    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4669b11d-cc77-431d-8f96-80e296a7e193-kube-api-access-mvslv" pod "4669b11d-cc77-431d-8f96-80e296a7e193" (UID: "4669b11d-cc77-431d-8f96-80e296a7e193"). InnerVolumeSpecName "kube-api-access-mvslv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:42.571711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:42.571676    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvslv\" (UniqueName: \"kubernetes.io/projected/4669b11d-cc77-431d-8f96-80e296a7e193-kube-api-access-mvslv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:39:43.183056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.183013    4519 generic.go:356] "Generic (PLEG): container finished" podID="4669b11d-cc77-431d-8f96-80e296a7e193" containerID="4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901" exitCode=0
Oct 30 20:39:43.183492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.183065    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"
Oct 30 20:39:43.183492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.183094    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz" event={"ID":"4669b11d-cc77-431d-8f96-80e296a7e193","Type":"ContainerDied","Data":"4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901"}
Oct 30 20:39:43.183492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.183150    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz" event={"ID":"4669b11d-cc77-431d-8f96-80e296a7e193","Type":"ContainerDied","Data":"d1d1b875f2ae7d6594b9f5372aaae7e045abee0c88a9c54013e1f907c0547c2f"}
Oct 30 20:39:43.183492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.183174    4519 scope.go:122] "RemoveContainer" containerID="4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901"
Oct 30 20:39:43.196755 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.196735    4519 scope.go:122] "RemoveContainer" containerID="4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901"
Oct 30 20:39:43.197055 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:39:43.197032    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901\": container with ID starting with 4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901 not found: ID does not exist" containerID="4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901"
Oct 30 20:39:43.197164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.197062    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901"} err="failed to get container status \"4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901\": rpc error: code = NotFound desc = could not find container \"4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901\": container with ID starting with 4679bf805d9dd1b4eb68cb1356994c8ed7b6b8e04bd60428e07a85a94c8cb901 not found: ID does not exist"
Oct 30 20:39:43.246368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.246342    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"]
Oct 30 20:39:43.258649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.258623    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hdfxz"]
Oct 30 20:39:43.610819 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:39:43.610742    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4669b11d-cc77-431d-8f96-80e296a7e193" path="/var/lib/kubelet/pods/4669b11d-cc77-431d-8f96-80e296a7e193/volumes"
Oct 30 20:40:02.219970 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:40:02.219924    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:40:32.222376 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:40:32.222330    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:40:45.297223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.297182    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"]
Oct 30 20:40:45.297657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.297583    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4669b11d-cc77-431d-8f96-80e296a7e193" containerName="manager"
Oct 30 20:40:45.297657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.297595    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4669b11d-cc77-431d-8f96-80e296a7e193" containerName="manager"
Oct 30 20:40:45.297657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.297606    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4088c15c-949b-4a90-a062-d5e341fcd06c" containerName="main"
Oct 30 20:40:45.297657 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.297612    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4088c15c-949b-4a90-a062-d5e341fcd06c" containerName="main"
Oct 30 20:40:45.297812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.297703    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4669b11d-cc77-431d-8f96-80e296a7e193" containerName="manager"
Oct 30 20:40:45.297812 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.297712    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4088c15c-949b-4a90-a062-d5e341fcd06c" containerName="main"
Oct 30 20:40:45.300682 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.300664    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.327601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.327573    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-465tx\""
Oct 30 20:40:45.329859 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.329835    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"]
Oct 30 20:40:45.408046 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.408011    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrpd\" (UniqueName: \"kubernetes.io/projected/e0c8bab4-b32c-40fe-9c93-2ffb88d812a2-kube-api-access-tlrpd\") pod \"kuadrant-operator-controller-manager-678d87766f-5lg26\" (UID: \"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.408214 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.408084    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0c8bab4-b32c-40fe-9c93-2ffb88d812a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5lg26\" (UID: \"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.508866 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.508824    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrpd\" (UniqueName: \"kubernetes.io/projected/e0c8bab4-b32c-40fe-9c93-2ffb88d812a2-kube-api-access-tlrpd\") pod \"kuadrant-operator-controller-manager-678d87766f-5lg26\" (UID: \"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.509057 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.508925    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0c8bab4-b32c-40fe-9c93-2ffb88d812a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5lg26\" (UID: \"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.509331 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.509307    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0c8bab4-b32c-40fe-9c93-2ffb88d812a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5lg26\" (UID: \"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.528385 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.528359    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrpd\" (UniqueName: \"kubernetes.io/projected/e0c8bab4-b32c-40fe-9c93-2ffb88d812a2-kube-api-access-tlrpd\") pod \"kuadrant-operator-controller-manager-678d87766f-5lg26\" (UID: \"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.613515 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.613396    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:45.949386 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:45.949357    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"]
Oct 30 20:40:45.951629 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:40:45.951599    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c8bab4_b32c_40fe_9c93_2ffb88d812a2.slice/crio-d5e441b2b3a28f57436c501e2c54b502287e3068a982dd975bd2d946d7cfec74 WatchSource:0}: Error finding container d5e441b2b3a28f57436c501e2c54b502287e3068a982dd975bd2d946d7cfec74: Status 404 returned error can't find the container with id d5e441b2b3a28f57436c501e2c54b502287e3068a982dd975bd2d946d7cfec74
Oct 30 20:40:46.497550 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:46.497501    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26" event={"ID":"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2","Type":"ContainerStarted","Data":"24a73dbf0117d5ffd2c86593da7d3b67f8973a114d1be9e11454a49d88e6a164"}
Oct 30 20:40:46.497964 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:46.497584    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:40:46.497964 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:46.497605    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26" event={"ID":"e0c8bab4-b32c-40fe-9c93-2ffb88d812a2","Type":"ContainerStarted","Data":"d5e441b2b3a28f57436c501e2c54b502287e3068a982dd975bd2d946d7cfec74"}
Oct 30 20:40:46.541388 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:46.541335    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26" podStartSLOduration=1.5413212920000001 podStartE2EDuration="1.541321292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:46.539211637 +0000 UTC m=+3971.039915373" watchObservedRunningTime="2026-10-30 20:40:46.541321292 +0000 UTC m=+3971.042025016"
Oct 30 20:40:57.505262 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:40:57.505225    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5lg26"
Oct 30 20:41:02.220099 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:41:02.220058    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:41:32.220309 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:41:32.220220    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:42:02.220260 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:42:02.220221    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:42:32.223871 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:42:32.223826    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:42:50.448615 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.448513    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"]
Oct 30 20:42:50.453048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.453010    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.472549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.472511    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"]
Oct 30 20:42:50.481563 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.481506    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-051802\"/\"kube-root-ca.crt\""
Oct 30 20:42:50.482743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.482712    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-051802\"/\"default-dockercfg-7mkb2\""
Oct 30 20:42:50.482907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.482772    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-051802\"/\"openshift-service-ca.crt\""
Oct 30 20:42:50.482907 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.482816    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-051802\"/\"simulator-single-main-051802-kserve-self-signed-certs\""
Oct 30 20:42:50.625576 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.625532    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j55m5\" (UniqueName: \"kubernetes.io/projected/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-kube-api-access-j55m5\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.625790 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.625678    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tmp-dir\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.625862 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.625821    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-model-cache\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.625910 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.625895    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-dshm\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.625962 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.625945    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-home\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.626003 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.625989    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tls-certs\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727098    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-model-cache\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727161    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-dshm\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727228 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727202    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-home\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727236    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tls-certs\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727272    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55m5\" (UniqueName: \"kubernetes.io/projected/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-kube-api-access-j55m5\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727376    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tmp-dir\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727721    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-home\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727851 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727833    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-model-cache\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.727917 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.727898    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tmp-dir\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.730209 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.730176    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-dshm\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.730381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.730244    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tls-certs\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.747130 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.747094    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55m5\" (UniqueName: \"kubernetes.io/projected/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-kube-api-access-j55m5\") pod \"simulator-single-main-051802-kserve-5c8bdc8674-6nf59\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") " pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:50.780996 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:50.780952    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:42:55.378017 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:42:55.377963    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"]
Oct 30 20:43:02.222288 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:43:02.222238    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:43:32.222483 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:43:32.222347    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:43:35.268044 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:43:35.268000    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod147dd77d_77e0_4bfc_8410_ba7db8a6cd40.slice/crio-cfbeff038935a1d32bd212a898b137c1897ec8b3ee5ff1fd3c087245b4907fa5 WatchSource:0}: Error finding container cfbeff038935a1d32bd212a898b137c1897ec8b3ee5ff1fd3c087245b4907fa5: Status 404 returned error can't find the container with id cfbeff038935a1d32bd212a898b137c1897ec8b3ee5ff1fd3c087245b4907fa5
Oct 30 20:43:35.479395 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:43:35.479347    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59" event={"ID":"147dd77d-77e0-4bfc-8410-ba7db8a6cd40","Type":"ContainerStarted","Data":"ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e"}
Oct 30 20:43:35.479610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:43:35.479409    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59" event={"ID":"147dd77d-77e0-4bfc-8410-ba7db8a6cd40","Type":"ContainerStarted","Data":"cfbeff038935a1d32bd212a898b137c1897ec8b3ee5ff1fd3c087245b4907fa5"}
Oct 30 20:43:35.519920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:43:35.519781    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59" podStartSLOduration=45.519763245 podStartE2EDuration="45.519763245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:35.518740578 +0000 UTC m=+4140.019444305" watchObservedRunningTime="2026-10-30 20:43:35.519763245 +0000 UTC m=+4140.020466981"
Oct 30 20:43:35.781598 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:43:35.781506    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:43:40.781418 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:43:40.781372    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:43:40.794199 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:43:40.794167    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:43:41.527813 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:43:41.527776    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:44:02.222612 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:44:02.222566    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:44:18.119389 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:18.119336    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"]
Oct 30 20:44:32.224332 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:44:32.224233    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:44:33.140937 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.140877    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59" podUID="147dd77d-77e0-4bfc-8410-ba7db8a6cd40" containerName="main" containerID="cri-o://ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e" gracePeriod=45
Oct 30 20:44:33.400163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.400078    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:44:33.555417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555368    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tls-certs\" (UniqueName: \"kubernetes.io/secret/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tls-certs\") pod \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") "
Oct 30 20:44:33.555627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555454    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-model-cache\") pod \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") "
Oct 30 20:44:33.555690 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555667    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-home\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-home\") pod \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") "
Oct 30 20:44:33.555743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555719    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-dshm\") pod \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") "
Oct 30 20:44:33.555743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555734    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-model-cache" pod "147dd77d-77e0-4bfc-8410-ba7db8a6cd40" (UID: "147dd77d-77e0-4bfc-8410-ba7db8a6cd40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:33.555849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555770    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tmp-dir\") pod \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") "
Oct 30 20:44:33.555849 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555805    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-kube-api-access-j55m5\" (UniqueName: \"kubernetes.io/projected/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-kube-api-access-j55m5\") pod \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\" (UID: \"147dd77d-77e0-4bfc-8410-ba7db8a6cd40\") "
Oct 30 20:44:33.555974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.555867    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-home" pod "147dd77d-77e0-4bfc-8410-ba7db8a6cd40" (UID: "147dd77d-77e0-4bfc-8410-ba7db8a6cd40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:33.556037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.556015    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tmp-dir" pod "147dd77d-77e0-4bfc-8410-ba7db8a6cd40" (UID: "147dd77d-77e0-4bfc-8410-ba7db8a6cd40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:33.556167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.556149    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:44:33.556167 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.556167    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:44:33.556296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.556176    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:44:33.557914 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.557878    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-dshm" pod "147dd77d-77e0-4bfc-8410-ba7db8a6cd40" (UID: "147dd77d-77e0-4bfc-8410-ba7db8a6cd40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:33.558035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.558015    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-kube-api-access-j55m5" pod "147dd77d-77e0-4bfc-8410-ba7db8a6cd40" (UID: "147dd77d-77e0-4bfc-8410-ba7db8a6cd40"). InnerVolumeSpecName "kube-api-access-j55m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:33.558085 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.558015    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tls-certs" pod "147dd77d-77e0-4bfc-8410-ba7db8a6cd40" (UID: "147dd77d-77e0-4bfc-8410-ba7db8a6cd40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:33.657722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.657620    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:44:33.657722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.657661    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:44:33.657722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.657671    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j55m5\" (UniqueName: \"kubernetes.io/projected/147dd77d-77e0-4bfc-8410-ba7db8a6cd40-kube-api-access-j55m5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:44:33.820332 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.820290    4519 generic.go:356] "Generic (PLEG): container finished" podID="147dd77d-77e0-4bfc-8410-ba7db8a6cd40" containerID="ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e" exitCode=0
Oct 30 20:44:33.820539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.820337    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"
Oct 30 20:44:33.820539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.820371    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59" event={"ID":"147dd77d-77e0-4bfc-8410-ba7db8a6cd40","Type":"ContainerDied","Data":"ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e"}
Oct 30 20:44:33.820539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.820418    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59" event={"ID":"147dd77d-77e0-4bfc-8410-ba7db8a6cd40","Type":"ContainerDied","Data":"cfbeff038935a1d32bd212a898b137c1897ec8b3ee5ff1fd3c087245b4907fa5"}
Oct 30 20:44:33.820539 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.820435    4519 scope.go:122] "RemoveContainer" containerID="ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e"
Oct 30 20:44:33.830005 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.829977    4519 scope.go:122] "RemoveContainer" containerID="ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e"
Oct 30 20:44:33.830312 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:44:33.830286    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e\": container with ID starting with ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e not found: ID does not exist" containerID="ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e"
Oct 30 20:44:33.830367 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.830322    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e"} err="failed to get container status \"ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e\": rpc error: code = NotFound desc = could not find container \"ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e\": container with ID starting with ab256502d6303132eacdf3ffb7d772f2c4761ce734c96a67d03804fc9c84cf4e not found: ID does not exist"
Oct 30 20:44:33.873938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.873884    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"]
Oct 30 20:44:33.885949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:33.885903    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-051802/simulator-single-main-051802-kserve-5c8bdc8674-6nf59"]
Oct 30 20:44:35.614099 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:35.614055    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147dd77d-77e0-4bfc-8410-ba7db8a6cd40" path="/var/lib/kubelet/pods/147dd77d-77e0-4bfc-8410-ba7db8a6cd40/volumes"
Oct 30 20:44:38.846514 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.846454    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"]
Oct 30 20:44:38.847134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.847108    4519 cpu_manager.go:414] "containerMap: removing container" podUID="147dd77d-77e0-4bfc-8410-ba7db8a6cd40" containerName="main"
Oct 30 20:44:38.847223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.847138    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147dd77d-77e0-4bfc-8410-ba7db8a6cd40" containerName="main"
Oct 30 20:44:38.847283 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.847262    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="147dd77d-77e0-4bfc-8410-ba7db8a6cd40" containerName="main"
Oct 30 20:44:38.851000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.850970    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:38.872634 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.872587    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"]
Oct 30 20:44:38.881189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.881151    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-674xl\""
Oct 30 20:44:38.881189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.881163    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8938eac6-kserve-self-signed-certs\""
Oct 30 20:44:38.885928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.885899    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 20:44:38.885928 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:38.885900    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 20:44:39.022499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.022423    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-model-cache\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.022499 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.022500    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tmp-dir\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.022737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.022623    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tls-certs\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.022737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.022661    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxwqv\" (UniqueName: \"kubernetes.io/projected/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-kube-api-access-lxwqv\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.022737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.022689    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-dshm\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.022868 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.022850    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-home\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.123830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.123705    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tmp-dir\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.123830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.123767    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tls-certs\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.123830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.123788    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxwqv\" (UniqueName: \"kubernetes.io/projected/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-kube-api-access-lxwqv\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.123830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.123812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-dshm\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.124266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.123884    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-home\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.124266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.123934    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-model-cache\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.124266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.124237    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tmp-dir\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.124443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.124339    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-model-cache\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.124443 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.124428    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-home\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.126361 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.126298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-dshm\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.126697 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.126670    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tls-certs\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.144371 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.144334    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxwqv\" (UniqueName: \"kubernetes.io/projected/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-kube-api-access-lxwqv\") pod \"e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") " pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.178190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.178143    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:39.531562 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.531526    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"]
Oct 30 20:44:39.533210 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:44:39.533172    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70ea3fb4_540c_4f79_bee7_8c43c0b88cfd.slice/crio-dfb093ae2f2a6713d4db3f98e0118cbb1afce4ed3ceb7def4d5893c022312915 WatchSource:0}: Error finding container dfb093ae2f2a6713d4db3f98e0118cbb1afce4ed3ceb7def4d5893c022312915: Status 404 returned error can't find the container with id dfb093ae2f2a6713d4db3f98e0118cbb1afce4ed3ceb7def4d5893c022312915
Oct 30 20:44:39.864446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.864339    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v" event={"ID":"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd","Type":"ContainerStarted","Data":"c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38"}
Oct 30 20:44:39.864446 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.864388    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v" event={"ID":"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd","Type":"ContainerStarted","Data":"dfb093ae2f2a6713d4db3f98e0118cbb1afce4ed3ceb7def4d5893c022312915"}
Oct 30 20:44:39.937622 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:39.937558    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v" podStartSLOduration=1.93754471 podStartE2EDuration="1.93754471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:39.932958985 +0000 UTC m=+4204.433662709" watchObservedRunningTime="2026-10-30 20:44:39.93754471 +0000 UTC m=+4204.438248434"
Oct 30 20:44:40.178284 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:40.178234    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:49.179206 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:49.179151    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:49.192381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:49.192341    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:49.972061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:49.972020    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:44:55.967543 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:55.967496    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"]
Oct 30 20:44:55.973985 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:55.973951    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:55.995204 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:55.995161    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"]
Oct 30 20:44:56.000009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:55.999713    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8938eac6-kserve-self-signed-certs\""
Oct 30 20:44:56.028575 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.028527    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllk5\" (UniqueName: \"kubernetes.io/projected/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-kube-api-access-fllk5\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.028811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.028607    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-dshm\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.028811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.028686    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tls-certs\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.028811 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.028745    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-model-cache\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.029001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.028811    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-home\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.029001 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.028863    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tmp-dir\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.130407 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.130354    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-dshm\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.130649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.130416    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tls-certs\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.130649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.130456    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-model-cache\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.130649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.130563    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-home\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.130649 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.130596    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tmp-dir\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.130977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.130666    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllk5\" (UniqueName: \"kubernetes.io/projected/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-kube-api-access-fllk5\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.130977 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.130964    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-model-cache\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.131090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.131014    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-home\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.131090 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.131032    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tmp-dir\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.132984 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.132955    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-dshm\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.133239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.133218    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tls-certs\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.158025 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.157526    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllk5\" (UniqueName: \"kubernetes.io/projected/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-kube-api-access-fllk5\") pod \"e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") " pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.296235 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.296142    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:44:56.643419 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:56.643297    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"]
Oct 30 20:44:56.649833 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:44:56.649797    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71d26d8f_4cd7_4c40_a2b1_d5a2315375aa.slice/crio-9af8f822dabb20f8cfc07bc956b0f9a45e8b9f4aeae8860cb3f4a8d2641a7f37 WatchSource:0}: Error finding container 9af8f822dabb20f8cfc07bc956b0f9a45e8b9f4aeae8860cb3f4a8d2641a7f37: Status 404 returned error can't find the container with id 9af8f822dabb20f8cfc07bc956b0f9a45e8b9f4aeae8860cb3f4a8d2641a7f37
Oct 30 20:44:57.005820 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:57.005774    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz" event={"ID":"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa","Type":"ContainerStarted","Data":"a6130f68717711689060e4fb42610eed3ac9d3f563ebf26d8cb9994e35194e8a"}
Oct 30 20:44:57.006281 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:57.005832    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz" event={"ID":"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa","Type":"ContainerStarted","Data":"9af8f822dabb20f8cfc07bc956b0f9a45e8b9f4aeae8860cb3f4a8d2641a7f37"}
Oct 30 20:44:57.049381 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:57.049315    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz" podStartSLOduration=2.049296872 podStartE2EDuration="2.049296872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:57.047322547 +0000 UTC m=+4221.548026273" watchObservedRunningTime="2026-10-30 20:44:57.049296872 +0000 UTC m=+4221.550000596"
Oct 30 20:44:57.297376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:44:57.297274    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:45:02.269316 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:45:02.269267    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:45:06.296290 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:06.296240    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:45:06.308962 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:06.308921    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:45:07.084972 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:07.084922    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:45:12.560266 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.560219    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"]
Oct 30 20:45:12.564145 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.564120    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.591656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.591599    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:45:12.591852 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.591675    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:45:12.595517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.595459    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"]
Oct 30 20:45:12.753750 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.753680    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.754006 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.753870    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcww\" (UniqueName: \"kubernetes.io/projected/b9ebcc8e-ef02-4549-8e09-4017de546f5c-kube-api-access-dxcww\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.754077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.753994    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.855104 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.855008    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcww\" (UniqueName: \"kubernetes.io/projected/b9ebcc8e-ef02-4549-8e09-4017de546f5c-kube-api-access-dxcww\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.855308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.855109    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.855308 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.855163    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.856134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.856106    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.857771 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.857744    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.875307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.875267    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcww\" (UniqueName: \"kubernetes.io/projected/b9ebcc8e-ef02-4549-8e09-4017de546f5c-kube-api-access-dxcww\") pod \"maas-api-revoke-sub-452b6d7b-sqsfx\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:12.879930 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:12.879881    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:13.209655 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:13.209609    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"]
Oct 30 20:45:13.224978 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:45:13.224937    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ebcc8e_ef02_4549_8e09_4017de546f5c.slice/crio-ca3979cee9f8675282aca92fffce85b3ab22a6f81f3360a780f650daf6643f95 WatchSource:0}: Error finding container ca3979cee9f8675282aca92fffce85b3ab22a6f81f3360a780f650daf6643f95: Status 404 returned error can't find the container with id ca3979cee9f8675282aca92fffce85b3ab22a6f81f3360a780f650daf6643f95
Oct 30 20:45:14.116239 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:14.116194    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx" event={"ID":"b9ebcc8e-ef02-4549-8e09-4017de546f5c","Type":"ContainerStarted","Data":"dfbf99b3f0d4e06ddc4c46a19e03f764449bb97cdb74615672bb93f9e22d6eac"}
Oct 30 20:45:14.116710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:14.116263    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx" event={"ID":"b9ebcc8e-ef02-4549-8e09-4017de546f5c","Type":"ContainerStarted","Data":"ca3979cee9f8675282aca92fffce85b3ab22a6f81f3360a780f650daf6643f95"}
Oct 30 20:45:15.127246 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:15.127203    4519 generic.go:356] "Generic (PLEG): container finished" podID="b9ebcc8e-ef02-4549-8e09-4017de546f5c" containerID="dfbf99b3f0d4e06ddc4c46a19e03f764449bb97cdb74615672bb93f9e22d6eac" exitCode=0
Oct 30 20:45:15.127954 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:15.127283    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx" event={"ID":"b9ebcc8e-ef02-4549-8e09-4017de546f5c","Type":"ContainerDied","Data":"dfbf99b3f0d4e06ddc4c46a19e03f764449bb97cdb74615672bb93f9e22d6eac"}
Oct 30 20:45:16.694026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.693985    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:16.781492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.780877    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ebcc8e-ef02-4549-8e09-4017de546f5c-kube-api-access-dxcww\" (UniqueName: \"kubernetes.io/projected/b9ebcc8e-ef02-4549-8e09-4017de546f5c-kube-api-access-dxcww\") pod \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") "
Oct 30 20:45:16.781492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.780993    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-openshift-ca\") pod \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") "
Oct 30 20:45:16.781492 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.781031    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-xks-ca\") pod \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\" (UID: \"b9ebcc8e-ef02-4549-8e09-4017de546f5c\") "
Oct 30 20:45:16.808077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.808014    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-openshift-ca" pod "b9ebcc8e-ef02-4549-8e09-4017de546f5c" (UID: "b9ebcc8e-ef02-4549-8e09-4017de546f5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:16.837234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.837180    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ebcc8e-ef02-4549-8e09-4017de546f5c-kube-api-access-dxcww" pod "b9ebcc8e-ef02-4549-8e09-4017de546f5c" (UID: "b9ebcc8e-ef02-4549-8e09-4017de546f5c"). InnerVolumeSpecName "kube-api-access-dxcww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:16.839280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.839234    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-xks-ca" pod "b9ebcc8e-ef02-4549-8e09-4017de546f5c" (UID: "b9ebcc8e-ef02-4549-8e09-4017de546f5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:45:16.884280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.884159    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxcww\" (UniqueName: \"kubernetes.io/projected/b9ebcc8e-ef02-4549-8e09-4017de546f5c-kube-api-access-dxcww\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:16.884280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.884216    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:16.884280 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:16.884232    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebcc8e-ef02-4549-8e09-4017de546f5c-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:17.145546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:17.144485    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx" event={"ID":"b9ebcc8e-ef02-4549-8e09-4017de546f5c","Type":"ContainerDied","Data":"ca3979cee9f8675282aca92fffce85b3ab22a6f81f3360a780f650daf6643f95"}
Oct 30 20:45:17.145546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:17.144597    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3979cee9f8675282aca92fffce85b3ab22a6f81f3360a780f650daf6643f95"
Oct 30 20:45:17.145546 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:17.144676    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"
Oct 30 20:45:23.779417 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:23.779366    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"]
Oct 30 20:45:29.091599 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:29.083891    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"]
Oct 30 20:45:32.247907 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:45:32.247858    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:45:38.800732 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:38.800664    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v" podUID="70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" containerName="main" containerID="cri-o://c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38" gracePeriod=45
Oct 30 20:45:38.961956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:38.961911    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v" podUID="70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.108:8000/ready\": dial tcp 10.133.1.108:8000: connect: connection refused"
Oct 30 20:45:39.069268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.069235    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:45:39.150627 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.150587    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-dshm\") pod \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") "
Oct 30 20:45:39.150842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.150655    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tls-certs\") pod \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") "
Oct 30 20:45:39.150842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.150676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-kube-api-access-lxwqv\" (UniqueName: \"kubernetes.io/projected/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-kube-api-access-lxwqv\") pod \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") "
Oct 30 20:45:39.150842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.150722    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-model-cache\") pod \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") "
Oct 30 20:45:39.150842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.150786    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-home\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-home\") pod \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") "
Oct 30 20:45:39.150842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.150841    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tmp-dir\") pod \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\" (UID: \"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd\") "
Oct 30 20:45:39.151230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.151111    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-model-cache" pod "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" (UID: "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:39.151230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.151139    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-home" pod "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" (UID: "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:39.151230 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.151191    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tmp-dir" pod "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" (UID: "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:39.151406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.151252    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:39.151406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.151273    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:39.153000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.152975    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-dshm" pod "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" (UID: "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:39.153140 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.153088    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-kube-api-access-lxwqv" pod "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" (UID: "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd"). InnerVolumeSpecName "kube-api-access-lxwqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:39.153140 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.153133    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tls-certs" pod "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" (UID: "70ea3fb4-540c-4f79-bee7-8c43c0b88cfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:45:39.252000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.251955    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:39.252000 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.251998    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:39.252227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.252013    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxwqv\" (UniqueName: \"kubernetes.io/projected/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-kube-api-access-lxwqv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:39.252227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.252024    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:39.365194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.365103    4519 generic.go:356] "Generic (PLEG): container finished" podID="70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" containerID="c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38" exitCode=0
Oct 30 20:45:39.365194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.365153    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"
Oct 30 20:45:39.365194 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.365166    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v" event={"ID":"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd","Type":"ContainerDied","Data":"c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38"}
Oct 30 20:45:39.365504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.365216    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v" event={"ID":"70ea3fb4-540c-4f79-bee7-8c43c0b88cfd","Type":"ContainerDied","Data":"dfb093ae2f2a6713d4db3f98e0118cbb1afce4ed3ceb7def4d5893c022312915"}
Oct 30 20:45:39.365504 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.365239    4519 scope.go:122] "RemoveContainer" containerID="c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38"
Oct 30 20:45:39.375668 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.375640    4519 scope.go:122] "RemoveContainer" containerID="c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38"
Oct 30 20:45:39.376036 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:45:39.376007    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38\": container with ID starting with c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38 not found: ID does not exist" containerID="c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38"
Oct 30 20:45:39.376108 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.376048    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38"} err="failed to get container status \"c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38\": rpc error: code = NotFound desc = could not find container \"c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38\": container with ID starting with c2ffe05a38c9cf53048b8890c540aa75032bf570695739c158c8c9fe88918f38 not found: ID does not exist"
Oct 30 20:45:39.416203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.416158    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"]
Oct 30 20:45:39.420249 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.420215    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8938eac6-kserve-6d888fdc6f-cf82v"]
Oct 30 20:45:39.611329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:39.611289    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" path="/var/lib/kubelet/pods/70ea3fb4-540c-4f79-bee7-8c43c0b88cfd/volumes"
Oct 30 20:45:44.153565 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.153512    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz" podUID="71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" containerName="main" containerID="cri-o://a6130f68717711689060e4fb42610eed3ac9d3f563ebf26d8cb9994e35194e8a" gracePeriod=45
Oct 30 20:45:44.398736 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.398684    4519 generic.go:356] "Generic (PLEG): container finished" podID="71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" containerID="a6130f68717711689060e4fb42610eed3ac9d3f563ebf26d8cb9994e35194e8a" exitCode=0
Oct 30 20:45:44.398931 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.398753    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz" event={"ID":"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa","Type":"ContainerDied","Data":"a6130f68717711689060e4fb42610eed3ac9d3f563ebf26d8cb9994e35194e8a"}
Oct 30 20:45:44.428060 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.428018    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:45:44.501956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.501917    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tls-certs\") pod \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") "
Oct 30 20:45:44.502172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.501974    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-kube-api-access-fllk5\" (UniqueName: \"kubernetes.io/projected/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-kube-api-access-fllk5\") pod \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") "
Oct 30 20:45:44.502172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.502005    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-home\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-home\") pod \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") "
Oct 30 20:45:44.502172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.502035    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tmp-dir\") pod \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") "
Oct 30 20:45:44.502172 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.502072    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-model-cache\") pod \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") "
Oct 30 20:45:44.502729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.502211    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-dshm\") pod \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\" (UID: \"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa\") "
Oct 30 20:45:44.502729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.502493    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-home" pod "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" (UID: "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:44.502729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.502495    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tmp-dir" pod "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" (UID: "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:44.502729 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.502585    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-model-cache" pod "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" (UID: "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:44.504759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.504719    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-dshm" pod "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" (UID: "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:44.504938 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.504897    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tls-certs" pod "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" (UID: "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:45:44.505011 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.504956    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-kube-api-access-fllk5" pod "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" (UID: "71d26d8f-4cd7-4c40-a2b1-d5a2315375aa"). InnerVolumeSpecName "kube-api-access-fllk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:44.603997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.603947    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:44.603997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.603991    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fllk5\" (UniqueName: \"kubernetes.io/projected/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-kube-api-access-fllk5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:44.603997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.604001    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:44.603997 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.604010    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:44.604299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.604020    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:44.604299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:44.604028    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:45:45.406639 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:45.406600    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz" event={"ID":"71d26d8f-4cd7-4c40-a2b1-d5a2315375aa","Type":"ContainerDied","Data":"9af8f822dabb20f8cfc07bc956b0f9a45e8b9f4aeae8860cb3f4a8d2641a7f37"}
Oct 30 20:45:45.407089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:45.406648    4519 scope.go:122] "RemoveContainer" containerID="a6130f68717711689060e4fb42610eed3ac9d3f563ebf26d8cb9994e35194e8a"
Oct 30 20:45:45.407089 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:45.406616    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"
Oct 30 20:45:45.460973 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:45.460936    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"]
Oct 30 20:45:45.474887 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:45.474839    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8938eac6-kserve-75796c594f-lmhzz"]
Oct 30 20:45:45.612174 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:45:45.612130    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" path="/var/lib/kubelet/pods/71d26d8f-4cd7-4c40-a2b1-d5a2315375aa/volumes"
Oct 30 20:46:02.221287 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:46:02.221242    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:46:31.750618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.750567    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"]
Oct 30 20:46:31.751183 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751157    4519 cpu_manager.go:414] "containerMap: removing container" podUID="71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" containerName="main"
Oct 30 20:46:31.751267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751187    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" containerName="main"
Oct 30 20:46:31.751267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751205    4519 cpu_manager.go:414] "containerMap: removing container" podUID="70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" containerName="main"
Oct 30 20:46:31.751267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751214    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" containerName="main"
Oct 30 20:46:31.751267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751226    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b9ebcc8e-ef02-4549-8e09-4017de546f5c" containerName="revoke-keys"
Oct 30 20:46:31.751267 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751234    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ebcc8e-ef02-4549-8e09-4017de546f5c" containerName="revoke-keys"
Oct 30 20:46:31.751571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751345    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ea3fb4-540c-4f79-bee7-8c43c0b88cfd" containerName="main"
Oct 30 20:46:31.751571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751360    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ebcc8e-ef02-4549-8e09-4017de546f5c" containerName="revoke-keys"
Oct 30 20:46:31.751571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.751378    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d26d8f-4cd7-4c40-a2b1-d5a2315375aa" containerName="main"
Oct 30 20:46:31.754783 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.754757    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.772929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.772890    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:31.772929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.772923    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\""
Oct 30 20:46:31.789034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.788987    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"]
Oct 30 20:46:31.881998 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.881950    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.882234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.882020    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66p4c\" (UniqueName: \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.882234 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.882134    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.983450 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.983395    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66p4c\" (UniqueName: \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.983726 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.983460    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.983726 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.983615    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.984363 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.984339    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:31.986257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:31.986229    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:32.010830 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:46:32.010737    4519 projected.go:196] Error preparing data for projected volume kube-api-access-66p4c for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:46:32.011007 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:46:32.010833    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c podName:a518162d-74bc-464f-b741-4cf50f0a2d20 nodeName:}" failed. No retries permitted until 2026-10-30 20:46:32.51081297 +0000 UTC m=+4317.011516689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-66p4c" (UniqueName: "kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c") pod "maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" (UID: "a518162d-74bc-464f-b741-4cf50f0a2d20") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:46:32.222660 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:46:32.222622    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:46:32.590329 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:32.590275    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66p4c\" (UniqueName: \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:32.605645 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:46:32.605596    4519 projected.go:196] Error preparing data for projected volume kube-api-access-66p4c for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:46:32.605887 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:46:32.605687    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c podName:a518162d-74bc-464f-b741-4cf50f0a2d20 nodeName:}" failed. No retries permitted until 2026-10-30 20:46:33.605666033 +0000 UTC m=+4318.106369738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-66p4c" (UniqueName: "kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c") pod "maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" (UID: "a518162d-74bc-464f-b741-4cf50f0a2d20") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:46:33.701832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:33.701785    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66p4c\" (UniqueName: \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:33.724494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:33.724429    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66p4c\" (UniqueName: \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\") pod \"maas-api-revoke-keys-e2e-ait-f5c2161a-v448g\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:33.868445 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:33.868389    4519 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" secret="" err="secret \"maas-api-cleanup-dockercfg-r8nlg\" not found"
Oct 30 20:46:33.868671 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:33.868494    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:34.172276 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:34.172228    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"]
Oct 30 20:46:34.187540 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:46:34.187506    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda518162d_74bc_464f_b741_4cf50f0a2d20.slice/crio-21b265569cd821916f4677b6ed1245cddd7bf071247401081b455065772beb5b WatchSource:0}: Error finding container 21b265569cd821916f4677b6ed1245cddd7bf071247401081b455065772beb5b: Status 404 returned error can't find the container with id 21b265569cd821916f4677b6ed1245cddd7bf071247401081b455065772beb5b
Oct 30 20:46:34.692159 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:34.692113    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" event={"ID":"a518162d-74bc-464f-b741-4cf50f0a2d20","Type":"ContainerStarted","Data":"56b3f6862ca2880640504dad7f327c9665c7c64c40f81f450fed6423c6535bf2"}
Oct 30 20:46:34.692353 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:34.692182    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" event={"ID":"a518162d-74bc-464f-b741-4cf50f0a2d20","Type":"ContainerStarted","Data":"21b265569cd821916f4677b6ed1245cddd7bf071247401081b455065772beb5b"}
Oct 30 20:46:34.692398 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:34.692351    4519 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" secret="" err="secret \"maas-api-cleanup-dockercfg-r8nlg\" not found"
Oct 30 20:46:34.734312 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:34.734183    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" podStartSLOduration=3.734168386 podStartE2EDuration="3.734168386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:34.731562818 +0000 UTC m=+4319.232266545" watchObservedRunningTime="2026-10-30 20:46:34.734168386 +0000 UTC m=+4319.234872110"
Oct 30 20:46:35.698848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:35.698811    4519 generic.go:356] "Generic (PLEG): container finished" podID="a518162d-74bc-464f-b741-4cf50f0a2d20" containerID="56b3f6862ca2880640504dad7f327c9665c7c64c40f81f450fed6423c6535bf2" exitCode=0
Oct 30 20:46:35.699049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:35.698899    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" event={"ID":"a518162d-74bc-464f-b741-4cf50f0a2d20","Type":"ContainerDied","Data":"56b3f6862ca2880640504dad7f327c9665c7c64c40f81f450fed6423c6535bf2"}
Oct 30 20:46:36.843944 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:36.843909    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:36.935586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:36.935543    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-xks-ca\") pod \"a518162d-74bc-464f-b741-4cf50f0a2d20\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") "
Oct 30 20:46:36.935799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:36.935629    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-openshift-ca\") pod \"a518162d-74bc-464f-b741-4cf50f0a2d20\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") "
Oct 30 20:46:36.935799 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:36.935655    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\" (UniqueName: \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\") pod \"a518162d-74bc-464f-b741-4cf50f0a2d20\" (UID: \"a518162d-74bc-464f-b741-4cf50f0a2d20\") "
Oct 30 20:46:36.936030 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:36.936004    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-openshift-ca" pod "a518162d-74bc-464f-b741-4cf50f0a2d20" (UID: "a518162d-74bc-464f-b741-4cf50f0a2d20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:36.938026 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:36.937995    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-xks-ca" pod "a518162d-74bc-464f-b741-4cf50f0a2d20" (UID: "a518162d-74bc-464f-b741-4cf50f0a2d20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:36.938136 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:36.938107    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c" pod "a518162d-74bc-464f-b741-4cf50f0a2d20" (UID: "a518162d-74bc-464f-b741-4cf50f0a2d20"). InnerVolumeSpecName "kube-api-access-66p4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:37.037069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:37.036958    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:46:37.037069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:37.036996    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66p4c\" (UniqueName: \"kubernetes.io/projected/a518162d-74bc-464f-b741-4cf50f0a2d20-kube-api-access-66p4c\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:46:37.037069 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:37.037008    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518162d-74bc-464f-b741-4cf50f0a2d20-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:46:37.712427 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:37.712392    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"
Oct 30 20:46:37.712623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:37.712424    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g" event={"ID":"a518162d-74bc-464f-b741-4cf50f0a2d20","Type":"ContainerDied","Data":"21b265569cd821916f4677b6ed1245cddd7bf071247401081b455065772beb5b"}
Oct 30 20:46:37.712623 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:37.712511    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b265569cd821916f4677b6ed1245cddd7bf071247401081b455065772beb5b"
Oct 30 20:46:43.489083 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:43.489033    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"]
Oct 30 20:46:43.501368 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:43.501313    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5c2161a-v448g"]
Oct 30 20:46:43.611189 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:46:43.611148    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a518162d-74bc-464f-b741-4cf50f0a2d20" path="/var/lib/kubelet/pods/a518162d-74bc-464f-b741-4cf50f0a2d20/volumes"
Oct 30 20:47:02.222249 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:47:02.222206    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:47:32.222530 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:47:32.222403    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:48:02.222197 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:48:02.222149    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:48:28.861586 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.861541    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"]
Oct 30 20:48:28.862164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.861989    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a518162d-74bc-464f-b741-4cf50f0a2d20" containerName="revoke-keys"
Oct 30 20:48:28.862164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.862004    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a518162d-74bc-464f-b741-4cf50f0a2d20" containerName="revoke-keys"
Oct 30 20:48:28.862164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.862079    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a518162d-74bc-464f-b741-4cf50f0a2d20" containerName="revoke-keys"
Oct 30 20:48:28.865227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.865203    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:28.896066 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.896022    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"]
Oct 30 20:48:28.896257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.896214    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhvdz\""
Oct 30 20:48:28.896257 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.896245    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:48:28.935412 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.935355    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:28.935643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.935443    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7n9v\" (UniqueName: \"kubernetes.io/projected/83578a69-3083-4f1a-8b85-40050c22a9e8-kube-api-access-m7n9v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:28.935696 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:28.935639    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.037101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.037053    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.037101 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.037104    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.037392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.037131    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7n9v\" (UniqueName: \"kubernetes.io/projected/83578a69-3083-4f1a-8b85-40050c22a9e8-kube-api-access-m7n9v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.037824 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.037794    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.039709 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.039683    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.060124 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.060083    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7n9v\" (UniqueName: \"kubernetes.io/projected/83578a69-3083-4f1a-8b85-40050c22a9e8-kube-api-access-m7n9v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.178952 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.178909    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:29.484746 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:29.484637    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"]
Oct 30 20:48:29.487713 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:48:29.487678    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83578a69_3083_4f1a_8b85_40050c22a9e8.slice/crio-597e862d21b512a435bb7047d6061817a731938bc9abfed54d9f511813591375 WatchSource:0}: Error finding container 597e862d21b512a435bb7047d6061817a731938bc9abfed54d9f511813591375: Status 404 returned error can't find the container with id 597e862d21b512a435bb7047d6061817a731938bc9abfed54d9f511813591375
Oct 30 20:48:30.356365 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:30.356320    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5" event={"ID":"83578a69-3083-4f1a-8b85-40050c22a9e8","Type":"ContainerStarted","Data":"9059f678e9162534b34063f97c7dd94fc04be8ace777d881a708b237b6899426"}
Oct 30 20:48:30.356810 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:30.356394    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5" event={"ID":"83578a69-3083-4f1a-8b85-40050c22a9e8","Type":"ContainerStarted","Data":"597e862d21b512a435bb7047d6061817a731938bc9abfed54d9f511813591375"}
Oct 30 20:48:30.411840 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:30.411777    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5" podStartSLOduration=2.411761847 podStartE2EDuration="2.411761847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:30.410584004 +0000 UTC m=+4434.911287729" watchObservedRunningTime="2026-10-30 20:48:30.411761847 +0000 UTC m=+4434.912465571"
Oct 30 20:48:31.363488 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:31.363421    4519 generic.go:356] "Generic (PLEG): container finished" podID="83578a69-3083-4f1a-8b85-40050c22a9e8" containerID="9059f678e9162534b34063f97c7dd94fc04be8ace777d881a708b237b6899426" exitCode=0
Oct 30 20:48:31.363924 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:31.363510    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5" event={"ID":"83578a69-3083-4f1a-8b85-40050c22a9e8","Type":"ContainerDied","Data":"9059f678e9162534b34063f97c7dd94fc04be8ace777d881a708b237b6899426"}
Oct 30 20:48:32.221709 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:48:32.221668    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:48:32.505035 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.505002    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:32.576041 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.576001    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-openshift-ca\") pod \"83578a69-3083-4f1a-8b85-40050c22a9e8\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") "
Oct 30 20:48:32.576268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.576055    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83578a69-3083-4f1a-8b85-40050c22a9e8-kube-api-access-m7n9v\" (UniqueName: \"kubernetes.io/projected/83578a69-3083-4f1a-8b85-40050c22a9e8-kube-api-access-m7n9v\") pod \"83578a69-3083-4f1a-8b85-40050c22a9e8\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") "
Oct 30 20:48:32.576268 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.576139    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-xks-ca\") pod \"83578a69-3083-4f1a-8b85-40050c22a9e8\" (UID: \"83578a69-3083-4f1a-8b85-40050c22a9e8\") "
Oct 30 20:48:32.576505 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.576452    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-openshift-ca" pod "83578a69-3083-4f1a-8b85-40050c22a9e8" (UID: "83578a69-3083-4f1a-8b85-40050c22a9e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:32.578364 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.578336    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83578a69-3083-4f1a-8b85-40050c22a9e8-kube-api-access-m7n9v" pod "83578a69-3083-4f1a-8b85-40050c22a9e8" (UID: "83578a69-3083-4f1a-8b85-40050c22a9e8"). InnerVolumeSpecName "kube-api-access-m7n9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:32.578770 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.578750    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-xks-ca" pod "83578a69-3083-4f1a-8b85-40050c22a9e8" (UID: "83578a69-3083-4f1a-8b85-40050c22a9e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:32.676958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.676912    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:48:32.676958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.676948    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7n9v\" (UniqueName: \"kubernetes.io/projected/83578a69-3083-4f1a-8b85-40050c22a9e8-kube-api-access-m7n9v\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:48:32.676958 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:32.676963    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83578a69-3083-4f1a-8b85-40050c22a9e8-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:48:33.376988 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:33.376955    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"
Oct 30 20:48:33.377187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:33.376945    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5" event={"ID":"83578a69-3083-4f1a-8b85-40050c22a9e8","Type":"ContainerDied","Data":"597e862d21b512a435bb7047d6061817a731938bc9abfed54d9f511813591375"}
Oct 30 20:48:33.377187 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:33.377086    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="597e862d21b512a435bb7047d6061817a731938bc9abfed54d9f511813591375"
Oct 30 20:48:46.817700 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:46.817648    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"]
Oct 30 20:48:46.829759 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:46.829714    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-72e6e5-nprn5"]
Oct 30 20:48:47.612890 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:47.612842    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83578a69-3083-4f1a-8b85-40050c22a9e8" path="/var/lib/kubelet/pods/83578a69-3083-4f1a-8b85-40050c22a9e8/volumes"
Oct 30 20:48:59.955656 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:59.955586    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-7vbs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:59.956369 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:48:59.956331    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-7vbs7" podUID="924bd04d-c63b-441d-a17b-fff0e36df522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:02.221707 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:49:02.221663    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:49:12.038886 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.038843    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"]
Oct 30 20:49:12.039313 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.039258    4519 cpu_manager.go:414] "containerMap: removing container" podUID="83578a69-3083-4f1a-8b85-40050c22a9e8" containerName="revoke-keys"
Oct 30 20:49:12.039313 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.039269    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83578a69-3083-4f1a-8b85-40050c22a9e8" containerName="revoke-keys"
Oct 30 20:49:12.039394 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.039358    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="83578a69-3083-4f1a-8b85-40050c22a9e8" containerName="revoke-keys"
Oct 30 20:49:12.042724 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.042697    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.057547 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.057509    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:49:12.069430 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.069389    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhvdz\""
Oct 30 20:49:12.072370 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.072328    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"]
Oct 30 20:49:12.087945 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.087898    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.088150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.087962    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbt7k\" (UniqueName: \"kubernetes.io/projected/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-kube-api-access-hbt7k\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.088150 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.088050    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.189154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.189094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.189154 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.189157    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbt7k\" (UniqueName: \"kubernetes.io/projected/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-kube-api-access-hbt7k\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.189517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.189192    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.189903 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.189879    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.191761 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.191733    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.209882 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.209840    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbt7k\" (UniqueName: \"kubernetes.io/projected/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-kube-api-access-hbt7k\") pod \"maas-api-revoke-sub-f03629db-s8m52\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.357079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.356966    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:12.676848 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:12.676801    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"]
Oct 30 20:49:12.692561 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:49:12.692522    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ebfdf6_19f7_4e0d_94c8_7fb5bbda0573.slice/crio-d5df7c5e6f29a7f6d2e61eb1da6ee33a87afad572da945fc9e369e62c30ef051 WatchSource:0}: Error finding container d5df7c5e6f29a7f6d2e61eb1da6ee33a87afad572da945fc9e369e62c30ef051: Status 404 returned error can't find the container with id d5df7c5e6f29a7f6d2e61eb1da6ee33a87afad572da945fc9e369e62c30ef051
Oct 30 20:49:13.625453 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:13.625410    4519 generic.go:356] "Generic (PLEG): container finished" podID="71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573" containerID="1748efb732a2bad01a79800e00e2a1209005a028455df1015d8d6956424472cd" exitCode=0
Oct 30 20:49:13.625956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:13.625502    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52" event={"ID":"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573","Type":"ContainerDied","Data":"1748efb732a2bad01a79800e00e2a1209005a028455df1015d8d6956424472cd"}
Oct 30 20:49:13.625956 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:13.625557    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52" event={"ID":"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573","Type":"ContainerStarted","Data":"d5df7c5e6f29a7f6d2e61eb1da6ee33a87afad572da945fc9e369e62c30ef051"}
Oct 30 20:49:14.782737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.782705    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:14.814125 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.814083    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-kube-api-access-hbt7k\" (UniqueName: \"kubernetes.io/projected/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-kube-api-access-hbt7k\") pod \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") "
Oct 30 20:49:14.814314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.814217    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-openshift-ca\") pod \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") "
Oct 30 20:49:14.814314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.814260    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-xks-ca\") pod \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\" (UID: \"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573\") "
Oct 30 20:49:14.814660 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.814632    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-openshift-ca" pod "71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573" (UID: "71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:14.816507 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.816477    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-xks-ca" pod "71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573" (UID: "71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:14.816624 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.816482    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-kube-api-access-hbt7k" pod "71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573" (UID: "71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573"). InnerVolumeSpecName "kube-api-access-hbt7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:14.915472 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.915428    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbt7k\" (UniqueName: \"kubernetes.io/projected/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-kube-api-access-hbt7k\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:14.915678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.915508    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:14.915678 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:14.915521    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:15.638223 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:15.638190    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52"
Oct 30 20:49:15.638410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:15.638190    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s8m52" event={"ID":"71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573","Type":"ContainerDied","Data":"d5df7c5e6f29a7f6d2e61eb1da6ee33a87afad572da945fc9e369e62c30ef051"}
Oct 30 20:49:15.638410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:15.638325    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5df7c5e6f29a7f6d2e61eb1da6ee33a87afad572da945fc9e369e62c30ef051"
Oct 30 20:49:29.075227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.075184    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"]
Oct 30 20:49:29.075662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.075608    4519 cpu_manager.go:414] "containerMap: removing container" podUID="71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573" containerName="revoke-keys"
Oct 30 20:49:29.075662 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.075633    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573" containerName="revoke-keys"
Oct 30 20:49:29.075743 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.075712    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573" containerName="revoke-keys"
Oct 30 20:49:29.083247 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.083206    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.104056 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.104001    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"]
Oct 30 20:49:29.112503 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.112440    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:49:29.112692 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.112522    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhvdz\""
Oct 30 20:49:29.171833 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.171775    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.172034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.171853    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.172034 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.171992    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftplb\" (UniqueName: \"kubernetes.io/projected/a4d5beb0-5302-49f9-ad0f-efbf33a78565-kube-api-access-ftplb\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.273061 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.273014    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftplb\" (UniqueName: \"kubernetes.io/projected/a4d5beb0-5302-49f9-ad0f-efbf33a78565-kube-api-access-ftplb\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.273299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.273085    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.273299 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.273126    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.273888 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.273857    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.275629 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.275599    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.293725 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.293688    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftplb\" (UniqueName: \"kubernetes.io/projected/a4d5beb0-5302-49f9-ad0f-efbf33a78565-kube-api-access-ftplb\") pod \"maas-api-revoke-sub-0165d7d2-mrhkq\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.400710 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.400619    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:29.707227 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.707188    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"]
Oct 30 20:49:29.709529 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:49:29.709496    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d5beb0_5302_49f9_ad0f_efbf33a78565.slice/crio-d4aa4ac43e696ddf750103807914b074155757354392e5adcdf9974fefb3a3b8 WatchSource:0}: Error finding container d4aa4ac43e696ddf750103807914b074155757354392e5adcdf9974fefb3a3b8: Status 404 returned error can't find the container with id d4aa4ac43e696ddf750103807914b074155757354392e5adcdf9974fefb3a3b8
Oct 30 20:49:29.730797 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:29.730753    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq" event={"ID":"a4d5beb0-5302-49f9-ad0f-efbf33a78565","Type":"ContainerStarted","Data":"d4aa4ac43e696ddf750103807914b074155757354392e5adcdf9974fefb3a3b8"}
Oct 30 20:49:30.741669 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:30.741628    4519 generic.go:356] "Generic (PLEG): container finished" podID="a4d5beb0-5302-49f9-ad0f-efbf33a78565" containerID="21c89a05eda73d509f5f6ca6d37bfe0d4a757acb29347e8ec1de23931f206154" exitCode=0
Oct 30 20:49:30.742190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:30.741717    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq" event={"ID":"a4d5beb0-5302-49f9-ad0f-efbf33a78565","Type":"ContainerDied","Data":"21c89a05eda73d509f5f6ca6d37bfe0d4a757acb29347e8ec1de23931f206154"}
Oct 30 20:49:31.889867 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:31.889834    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:31.997314 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:31.997269    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d5beb0-5302-49f9-ad0f-efbf33a78565-kube-api-access-ftplb\" (UniqueName: \"kubernetes.io/projected/a4d5beb0-5302-49f9-ad0f-efbf33a78565-kube-api-access-ftplb\") pod \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") "
Oct 30 20:49:31.997567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:31.997420    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-openshift-ca\") pod \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") "
Oct 30 20:49:31.997567 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:31.997449    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-xks-ca\") pod \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\" (UID: \"a4d5beb0-5302-49f9-ad0f-efbf33a78565\") "
Oct 30 20:49:31.997920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:31.997856    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-openshift-ca" pod "a4d5beb0-5302-49f9-ad0f-efbf33a78565" (UID: "a4d5beb0-5302-49f9-ad0f-efbf33a78565"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:31.999764 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:31.999735    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d5beb0-5302-49f9-ad0f-efbf33a78565-kube-api-access-ftplb" pod "a4d5beb0-5302-49f9-ad0f-efbf33a78565" (UID: "a4d5beb0-5302-49f9-ad0f-efbf33a78565"). InnerVolumeSpecName "kube-api-access-ftplb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:31.999834 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:31.999777    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-xks-ca" pod "a4d5beb0-5302-49f9-ad0f-efbf33a78565" (UID: "a4d5beb0-5302-49f9-ad0f-efbf33a78565"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:32.099164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:32.099106    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftplb\" (UniqueName: \"kubernetes.io/projected/a4d5beb0-5302-49f9-ad0f-efbf33a78565-kube-api-access-ftplb\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:32.099164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:32.099151    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:32.099164 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:32.099162    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d5beb0-5302-49f9-ad0f-efbf33a78565-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:32.221376 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:49:32.221328    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:49:32.756356 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:32.756315    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq" event={"ID":"a4d5beb0-5302-49f9-ad0f-efbf33a78565","Type":"ContainerDied","Data":"d4aa4ac43e696ddf750103807914b074155757354392e5adcdf9974fefb3a3b8"}
Oct 30 20:49:32.756589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:32.756361    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mrhkq"
Oct 30 20:49:32.756589 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:32.756373    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4aa4ac43e696ddf750103807914b074155757354392e5adcdf9974fefb3a3b8"
Oct 30 20:49:40.073744 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.073693    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"]
Oct 30 20:49:40.074335 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.074313    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a4d5beb0-5302-49f9-ad0f-efbf33a78565" containerName="revoke-keys"
Oct 30 20:49:40.074390 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.074339    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d5beb0-5302-49f9-ad0f-efbf33a78565" containerName="revoke-keys"
Oct 30 20:49:40.074483 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.074454    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d5beb0-5302-49f9-ad0f-efbf33a78565" containerName="revoke-keys"
Oct 30 20:49:40.078396 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.078367    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.103245 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.103198    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"]
Oct 30 20:49:40.106437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.106400    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:49:40.106437 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.106418    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhvdz\""
Oct 30 20:49:40.182402 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.182354    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtpq9\" (UniqueName: \"kubernetes.io/projected/0080972b-ac82-4aa2-93e8-aa3a457b513d-kube-api-access-rtpq9\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.182643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.182487    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.182643 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.182561    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.283922 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.283877    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.284132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.283948    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtpq9\" (UniqueName: \"kubernetes.io/projected/0080972b-ac82-4aa2-93e8-aa3a457b513d-kube-api-access-rtpq9\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.284132 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.284014    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.284659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.284635    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.286929 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.286903    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.307144 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.307103    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtpq9\" (UniqueName: \"kubernetes.io/projected/0080972b-ac82-4aa2-93e8-aa3a457b513d-kube-api-access-rtpq9\") pod \"maas-api-revoke-sub-6b19338f-mnzbj\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.397037 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.396933    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:40.704748 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.704698    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"]
Oct 30 20:49:40.721774 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:49:40.721709    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0080972b_ac82_4aa2_93e8_aa3a457b513d.slice/crio-b74757dde4dabdd4a2d260610847b2b6323bffb0f05d562c25ea9310baae3c0a WatchSource:0}: Error finding container b74757dde4dabdd4a2d260610847b2b6323bffb0f05d562c25ea9310baae3c0a: Status 404 returned error can't find the container with id b74757dde4dabdd4a2d260610847b2b6323bffb0f05d562c25ea9310baae3c0a
Oct 30 20:49:40.806328 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:40.806288    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj" event={"ID":"0080972b-ac82-4aa2-93e8-aa3a457b513d","Type":"ContainerStarted","Data":"b74757dde4dabdd4a2d260610847b2b6323bffb0f05d562c25ea9310baae3c0a"}
Oct 30 20:49:41.817497 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:41.817379    4519 generic.go:356] "Generic (PLEG): container finished" podID="0080972b-ac82-4aa2-93e8-aa3a457b513d" containerID="c17c48bb7d0aaf9c7e559210d02d4239ba01a386fe959578cd13e5b93879da66" exitCode=0
Oct 30 20:49:41.817898 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:41.817484    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj" event={"ID":"0080972b-ac82-4aa2-93e8-aa3a457b513d","Type":"ContainerDied","Data":"c17c48bb7d0aaf9c7e559210d02d4239ba01a386fe959578cd13e5b93879da66"}
Oct 30 20:49:42.965202 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:42.965159    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:43.113982 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.113874    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-xks-ca\") pod \"0080972b-ac82-4aa2-93e8-aa3a457b513d\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") "
Oct 30 20:49:43.113982 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.113965    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-openshift-ca\") pod \"0080972b-ac82-4aa2-93e8-aa3a457b513d\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") "
Oct 30 20:49:43.114203 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.114033    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0080972b-ac82-4aa2-93e8-aa3a457b513d-kube-api-access-rtpq9\" (UniqueName: \"kubernetes.io/projected/0080972b-ac82-4aa2-93e8-aa3a457b513d-kube-api-access-rtpq9\") pod \"0080972b-ac82-4aa2-93e8-aa3a457b513d\" (UID: \"0080972b-ac82-4aa2-93e8-aa3a457b513d\") "
Oct 30 20:49:43.114423 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.114387    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-openshift-ca" pod "0080972b-ac82-4aa2-93e8-aa3a457b513d" (UID: "0080972b-ac82-4aa2-93e8-aa3a457b513d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:43.116190 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.116162    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-xks-ca" pod "0080972b-ac82-4aa2-93e8-aa3a457b513d" (UID: "0080972b-ac82-4aa2-93e8-aa3a457b513d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:43.116326 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.116217    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0080972b-ac82-4aa2-93e8-aa3a457b513d-kube-api-access-rtpq9" pod "0080972b-ac82-4aa2-93e8-aa3a457b513d" (UID: "0080972b-ac82-4aa2-93e8-aa3a457b513d"). InnerVolumeSpecName "kube-api-access-rtpq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:43.215631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.215577    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-xks-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:43.215631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.215615    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0080972b-ac82-4aa2-93e8-aa3a457b513d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:43.215631 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.215634    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtpq9\" (UniqueName: \"kubernetes.io/projected/0080972b-ac82-4aa2-93e8-aa3a457b513d-kube-api-access-rtpq9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Oct 30 20:49:43.837096 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.837057    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj"
Oct 30 20:49:43.837294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.837097    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnzbj" event={"ID":"0080972b-ac82-4aa2-93e8-aa3a457b513d","Type":"ContainerDied","Data":"b74757dde4dabdd4a2d260610847b2b6323bffb0f05d562c25ea9310baae3c0a"}
Oct 30 20:49:43.837294 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:49:43.837175    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b74757dde4dabdd4a2d260610847b2b6323bffb0f05d562c25ea9310baae3c0a"
Oct 30 20:50:02.222311 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:50:02.222266    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:50:02.274919 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:02.274882    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vmkgb_951ab11b-0442-4de2-93d8-5d4e09759ac7/manager/0.log"
Oct 30 20:50:02.776854 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:02.776818    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-t6gkp_d9a4b1c7-eb14-43dd-9b71-325a66bef4f8/manager/0.log"
Oct 30 20:50:03.041659 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:03.041558    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f86dcf78c-2cssg_54c18634-113b-4c35-aef2-29d15f9b545a/manager/0.log"
Oct 30 20:50:03.474571 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:03.474531    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xxgh4_67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb/manager/0.log"
Oct 30 20:50:03.694549 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:03.694415    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5qgtc_b2e9940f-4bb3-44c9-9dad-d71f47e4d627/proxy/0.log"
Oct 30 20:50:04.839808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:04.839776    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-mrhkq_a4d5beb0-5302-49f9-ad0f-efbf33a78565/revoke-keys/0.log"
Oct 30 20:50:05.218313 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:05.218285    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-sqsfx_b9ebcc8e-ef02-4549-8e09-4017de546f5c/revoke-keys/0.log"
Oct 30 20:50:05.403808 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:05.403770    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-mnzbj_0080972b-ac82-4aa2-93e8-aa3a457b513d/revoke-keys/0.log"
Oct 30 20:50:05.776610 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:05.776578    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s8m52_71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573/revoke-keys/0.log"
Oct 30 20:50:05.959079 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:05.959045    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-m7fmb_88f79d2d-2552-4139-8fef-c2af42193e46/postgres/0.log"
Oct 30 20:50:07.750853 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:07.750801    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qf89_f821e59d-1602-4bf8-9167-2ca51e9a8be4/manager/0.log"
Oct 30 20:50:07.970179 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:07.970144    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v25f9_777962be-d1c2-4e8d-8aa6-fe0d47c8a514/manager/0.log"
Oct 30 20:50:08.370095 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:08.370058    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5lg26_e0c8bab4-b32c-40fe-9c93-2ffb88d812a2/manager/0.log"
Oct 30 20:50:08.787772 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:08.787733    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4ct7c_12136fec-aa81-4940-adae-bf60070d3589/manager/0.log"
Oct 30 20:50:09.541260 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:09.541229    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-p78cx_6e04538a-4039-4b46-aba2-5fe298998516/istio-proxy/0.log"
Oct 30 20:50:10.578994 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:10.578955    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-7vbs7_924bd04d-c63b-441d-a17b-fff0e36df522/istio-proxy/1.log"
Oct 30 20:50:10.778376 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:10.778341    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-88mz8_0cf4fb75-5c06-41d1-904b-a1b6f0784eb4/istio-proxy/0.log"
Oct 30 20:50:10.973987 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:10.973949    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-8zcb4_24bb6760-98c6-4839-bfe7-6a6a5d42dca6/istio-proxy/0.log"
Oct 30 20:50:12.329050 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:12.329007    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22_cdd067e2-0384-46e0-b85f-4580b7a107fe/istio-proxy/0.log"
Oct 30 20:50:13.366236 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:13.366202    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-7vbs7_924bd04d-c63b-441d-a17b-fff0e36df522/istio-proxy/0.log"
Oct 30 20:50:17.029835 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:17.029793    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"]
Oct 30 20:50:17.041804 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:17.041763    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqsfx"]
Oct 30 20:50:17.611593 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:17.611553    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ebcc8e-ef02-4549-8e09-4017de546f5c" path="/var/lib/kubelet/pods/b9ebcc8e-ef02-4549-8e09-4017de546f5c/volumes"
Oct 30 20:50:25.716806 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:25.716726    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vmkgb_951ab11b-0442-4de2-93d8-5d4e09759ac7/manager/0.log"
Oct 30 20:50:26.125002 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:26.124912    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-t6gkp_d9a4b1c7-eb14-43dd-9b71-325a66bef4f8/manager/0.log"
Oct 30 20:50:26.354070 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:26.354031    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f86dcf78c-2cssg_54c18634-113b-4c35-aef2-29d15f9b545a/manager/0.log"
Oct 30 20:50:26.752548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:26.752509    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xxgh4_67469f7a-1c81-4e4e-9ab6-ddbd86b3d6fb/manager/0.log"
Oct 30 20:50:26.966605 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:26.966565    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5qgtc_b2e9940f-4bb3-44c9-9dad-d71f47e4d627/proxy/0.log"
Oct 30 20:50:27.888737 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:27.888698    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-mrhkq_a4d5beb0-5302-49f9-ad0f-efbf33a78565/revoke-keys/0.log"
Oct 30 20:50:28.290906 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:28.290870    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-mnzbj_0080972b-ac82-4aa2-93e8-aa3a457b513d/revoke-keys/0.log"
Oct 30 20:50:28.687974 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:28.687944    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s8m52_71ebfdf6-19f7-4e0d-94c8-7fb5bbda0573/revoke-keys/0.log"
Oct 30 20:50:28.884570 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:28.884535    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-m7fmb_88f79d2d-2552-4139-8fef-c2af42193e46/postgres/0.log"
Oct 30 20:50:30.263181 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:30.263128    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qf89_f821e59d-1602-4bf8-9167-2ca51e9a8be4/manager/0.log"
Oct 30 20:50:30.479307 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:30.479271    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v25f9_777962be-d1c2-4e8d-8aa6-fe0d47c8a514/manager/0.log"
Oct 30 20:50:30.878796 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:30.878753    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5lg26_e0c8bab4-b32c-40fe-9c93-2ffb88d812a2/manager/0.log"
Oct 30 20:50:31.302805 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:31.302769    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4ct7c_12136fec-aa81-4940-adae-bf60070d3589/manager/0.log"
Oct 30 20:50:31.881722 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:31.881686    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-p78cx_6e04538a-4039-4b46-aba2-5fe298998516/istio-proxy/0.log"
Oct 30 20:50:32.221684 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:50:32.221627    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:50:32.849554 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:32.849519    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-7vbs7_924bd04d-c63b-441d-a17b-fff0e36df522/istio-proxy/1.log"
Oct 30 20:50:33.048434 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:33.048401    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-88mz8_0cf4fb75-5c06-41d1-904b-a1b6f0784eb4/istio-proxy/0.log"
Oct 30 20:50:33.248883 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:33.248838    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-8zcb4_24bb6760-98c6-4839-bfe7-6a6a5d42dca6/istio-proxy/0.log"
Oct 30 20:50:34.627756 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:50:34.627720    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22_cdd067e2-0384-46e0-b85f-4580b7a107fe/istio-proxy/0.log"
Oct 30 20:51:02.221929 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:51:02.221883    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:51:12.268936 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:12.268895    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f86dcf78c-2cssg_54c18634-113b-4c35-aef2-29d15f9b545a/manager/0.log"
Oct 30 20:51:12.565410 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:12.565320    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f86dcf78c-2cssg_54c18634-113b-4c35-aef2-29d15f9b545a/manager/0.log"
Oct 30 20:51:32.222403 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:51:32.222354    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:51:46.828080 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:46.828037    4519 scope.go:122] "RemoveContainer" containerID="dfbf99b3f0d4e06ddc4c46a19e03f764449bb97cdb74615672bb93f9e22d6eac"
Oct 30 20:51:56.297510 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:56.297373    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f86dcf78c-2cssg_54c18634-113b-4c35-aef2-29d15f9b545a/manager/0.log"
Oct 30 20:51:56.574733 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:56.574638    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f86dcf78c-2cssg_54c18634-113b-4c35-aef2-29d15f9b545a/manager/0.log"
Oct 30 20:51:57.794832 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.794791    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b7snw/must-gather-tlbb8"]
Oct 30 20:51:57.795456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.795425    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0080972b-ac82-4aa2-93e8-aa3a457b513d" containerName="revoke-keys"
Oct 30 20:51:57.795608 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.795457    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0080972b-ac82-4aa2-93e8-aa3a457b513d" containerName="revoke-keys"
Oct 30 20:51:57.795689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.795606    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0080972b-ac82-4aa2-93e8-aa3a457b513d" containerName="revoke-keys"
Oct 30 20:51:57.799031 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.799006    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:57.814009 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.813972    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b7snw\"/\"default-dockercfg-965f6\""
Oct 30 20:51:57.826781 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.826704    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7snw\"/\"kube-root-ca.crt\""
Oct 30 20:51:57.829127 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.827157    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7snw\"/\"openshift-service-ca.crt\""
Oct 30 20:51:57.841830 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.841783    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/must-gather-tlbb8"]
Oct 30 20:51:57.893049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.893006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktgt\" (UniqueName: \"kubernetes.io/projected/168af5ce-1860-4882-a3d8-f89725bc703b-kube-api-access-7ktgt\") pod \"must-gather-tlbb8\" (UID: \"168af5ce-1860-4882-a3d8-f89725bc703b\") " pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:57.893049 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.893052    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/168af5ce-1860-4882-a3d8-f89725bc703b-must-gather-output\") pod \"must-gather-tlbb8\" (UID: \"168af5ce-1860-4882-a3d8-f89725bc703b\") " pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:57.993601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.993553    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktgt\" (UniqueName: \"kubernetes.io/projected/168af5ce-1860-4882-a3d8-f89725bc703b-kube-api-access-7ktgt\") pod \"must-gather-tlbb8\" (UID: \"168af5ce-1860-4882-a3d8-f89725bc703b\") " pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:57.993601 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.993601    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/168af5ce-1860-4882-a3d8-f89725bc703b-must-gather-output\") pod \"must-gather-tlbb8\" (UID: \"168af5ce-1860-4882-a3d8-f89725bc703b\") " pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:57.993923 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:57.993907    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/168af5ce-1860-4882-a3d8-f89725bc703b-must-gather-output\") pod \"must-gather-tlbb8\" (UID: \"168af5ce-1860-4882-a3d8-f89725bc703b\") " pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:58.014711 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:58.014672    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktgt\" (UniqueName: \"kubernetes.io/projected/168af5ce-1860-4882-a3d8-f89725bc703b-kube-api-access-7ktgt\") pod \"must-gather-tlbb8\" (UID: \"168af5ce-1860-4882-a3d8-f89725bc703b\") " pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:58.141528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:58.141399    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/must-gather-tlbb8"
Oct 30 20:51:58.443920 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:58.443888    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/must-gather-tlbb8"]
Oct 30 20:51:58.445705 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:51:58.445663    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod168af5ce_1860_4882_a3d8_f89725bc703b.slice/crio-6a8e2d706c5dcaa25cf29962c2d20f67529ad1d086c82588efbf182a0ff22b86 WatchSource:0}: Error finding container 6a8e2d706c5dcaa25cf29962c2d20f67529ad1d086c82588efbf182a0ff22b86: Status 404 returned error can't find the container with id 6a8e2d706c5dcaa25cf29962c2d20f67529ad1d086c82588efbf182a0ff22b86
Oct 30 20:51:58.447680 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:58.447656    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:51:58.623788 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:58.623745    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/must-gather-tlbb8" event={"ID":"168af5ce-1860-4882-a3d8-f89725bc703b","Type":"ContainerStarted","Data":"6a8e2d706c5dcaa25cf29962c2d20f67529ad1d086c82588efbf182a0ff22b86"}
Oct 30 20:51:59.632134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:59.632028    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/must-gather-tlbb8" event={"ID":"168af5ce-1860-4882-a3d8-f89725bc703b","Type":"ContainerStarted","Data":"2ea329f1a865279b1ed22486276c55efbf95c2c5c9cefb883b8f3e33c3694dda"}
Oct 30 20:51:59.632134 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:59.632083    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/must-gather-tlbb8" event={"ID":"168af5ce-1860-4882-a3d8-f89725bc703b","Type":"ContainerStarted","Data":"206d597f75c6184446aa7fb81e6064fa675094af3f98f694b9b0dfc5a80e53e4"}
Oct 30 20:51:59.671525 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:51:59.671428    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b7snw/must-gather-tlbb8" podStartSLOduration=1.8970015249999999 podStartE2EDuration="2.671408295s" totalImagesPullingTime="774.40677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:59.670016333 +0000 UTC m=+4644.170720058" watchObservedRunningTime="2026-10-30 20:51:59.671408295 +0000 UTC m=+4644.172112019"
Oct 30 20:52:02.223339 ip-10-0-133-26 kubenswrapper[4519]: E1030 20:52:02.223297    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kstg" containerName="extractor"
Oct 30 20:52:02.302842 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:02.302800    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-htpxk_c16bcfc9-20dd-4a12-ab7b-79dcbcb12557/global-pull-secret-syncer/0.log"
Oct 30 20:52:03.661487 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:03.655458    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zh9d6_0403f76e-6189-4e37-bcb0-b8cecd653283/konnectivity-agent/0.log"
Oct 30 20:52:03.995494 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:03.990956    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-26.ec2.internal_d8c3ec59a3a8e5471fc2a192c7750dd6/haproxy/0.log"
Oct 30 20:52:10.820594 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:10.820547    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qf89_f821e59d-1602-4bf8-9167-2ca51e9a8be4/manager/0.log"
Oct 30 20:52:10.947490 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:10.946758    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v25f9_777962be-d1c2-4e8d-8aa6-fe0d47c8a514/manager/0.log"
Oct 30 20:52:11.223814 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:11.223770    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5lg26_e0c8bab4-b32c-40fe-9c93-2ffb88d812a2/manager/0.log"
Oct 30 20:52:11.585820 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:11.585697    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4ct7c_12136fec-aa81-4940-adae-bf60070d3589/manager/0.log"
Oct 30 20:52:12.467392 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.467344    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"]
Oct 30 20:52:12.474632 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.474598    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.499581 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.495812    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"]
Oct 30 20:52:12.576301 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.576253    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-sys\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.576689 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.576626    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-proc\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.576924 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.576909    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-lib-modules\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.577048 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.577037    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2cd\" (UniqueName: \"kubernetes.io/projected/f62988f0-456b-4850-9e80-1c173865b3f8-kube-api-access-lr2cd\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.577193 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.577181    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-podres\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678426 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678378    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-proc\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678500    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-lib-modules\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678530    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2cd\" (UniqueName: \"kubernetes.io/projected/f62988f0-456b-4850-9e80-1c173865b3f8-kube-api-access-lr2cd\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678559    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-proc\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678572    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-podres\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678625    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-podres\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678651 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678631    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-lib-modules\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678676    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-sys\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.678980 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.678743    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f62988f0-456b-4850-9e80-1c173865b3f8-sys\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.698535 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.698454    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2cd\" (UniqueName: \"kubernetes.io/projected/f62988f0-456b-4850-9e80-1c173865b3f8-kube-api-access-lr2cd\") pod \"perf-node-gather-daemonset-mc622\" (UID: \"f62988f0-456b-4850-9e80-1c173865b3f8\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:12.793374 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:12.793280    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:13.131745 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:13.131714    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"]
Oct 30 20:52:13.134587 ip-10-0-133-26 kubenswrapper[4519]: W1030 20:52:13.134547    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf62988f0_456b_4850_9e80_1c173865b3f8.slice/crio-977f5489d7a1bf934c52ce6bf7f27149cb2cf5e3f6f2b9a85d0232ad207ab036 WatchSource:0}: Error finding container 977f5489d7a1bf934c52ce6bf7f27149cb2cf5e3f6f2b9a85d0232ad207ab036: Status 404 returned error can't find the container with id 977f5489d7a1bf934c52ce6bf7f27149cb2cf5e3f6f2b9a85d0232ad207ab036
Oct 30 20:52:13.835379 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:13.835334    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622" event={"ID":"f62988f0-456b-4850-9e80-1c173865b3f8","Type":"ContainerStarted","Data":"392b7eb5c338dad9e149276ae6ef4aba77b333d2ca4c08352fd6c9f3bcd74e33"}
Oct 30 20:52:13.835850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:13.835401    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622" event={"ID":"f62988f0-456b-4850-9e80-1c173865b3f8","Type":"ContainerStarted","Data":"977f5489d7a1bf934c52ce6bf7f27149cb2cf5e3f6f2b9a85d0232ad207ab036"}
Oct 30 20:52:13.835850 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:13.835493    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:13.875807 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:13.875720    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622" podStartSLOduration=1.8756980159999999 podStartE2EDuration="1.875698016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:13.873147718 +0000 UTC m=+4658.373851445" watchObservedRunningTime="2026-10-30 20:52:13.875698016 +0000 UTC m=+4658.376401743"
Oct 30 20:52:14.995587 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:14.995545    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrvbg_951e08f8-4211-4fd9-b759-1ebd5a2ccdc6/node-exporter/0.log"
Oct 30 20:52:15.117618 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:15.117582    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrvbg_951e08f8-4211-4fd9-b759-1ebd5a2ccdc6/kube-rbac-proxy/0.log"
Oct 30 20:52:15.240520 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:15.240487    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mrvbg_951e08f8-4211-4fd9-b759-1ebd5a2ccdc6/init-textfile/0.log"
Oct 30 20:52:16.953528 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:16.953456    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xfn9z_6ed79173-91c5-49da-9e0c-4818d7ad75ee/prometheus-operator/0.log"
Oct 30 20:52:17.068548 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:17.068510    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xfn9z_6ed79173-91c5-49da-9e0c-4818d7ad75ee/kube-rbac-proxy/0.log"
Oct 30 20:52:17.199024 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:17.198989    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2v84p_bdf5a7ee-c3fa-4650-b424-9fdb15c7e742/prometheus-operator-admission-webhook/0.log"
Oct 30 20:52:19.854163 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:19.854102    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-mc622"
Oct 30 20:52:22.010358 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:22.010317    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-644b5886bf-6qghr_f5ef3fe4-5e10-4fa7-9b73-eb5004650f06/console/0.log"
Oct 30 20:52:22.146517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:22.146449    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-kqqc8_f28b4d2c-a0a6-4ca0-bec4-f17c563b0de5/download-server/0.log"
Oct 30 20:52:25.973406 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:25.973364    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5wrbn_7aac5a86-e546-4160-8867-5610087a1b42/dns/0.log"
Oct 30 20:52:26.093022 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:26.092985    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5wrbn_7aac5a86-e546-4160-8867-5610087a1b42/kube-rbac-proxy/0.log"
Oct 30 20:52:26.095077 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:26.095047    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-p78cx_6e04538a-4039-4b46-aba2-5fe298998516/istio-proxy/0.log"
Oct 30 20:52:26.967650 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:26.967608    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qmwpk_5a47a9c2-1815-4bd6-b1bd-eb047f1cf679/dns-node-resolver/0.log"
Oct 30 20:52:28.111296 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:28.111253    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b665c84d-sdhgp_bd2c8fff-0ca4-45b8-a9a8-1eaeadcf2c2c/registry/0.log"
Oct 30 20:52:28.547817 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:28.547774    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mb5hn_adfd71eb-3976-4e6f-8bf2-182a1c9f22b5/node-ca/0.log"
Oct 30 20:52:28.941480 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:28.941425    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-7vbs7_924bd04d-c63b-441d-a17b-fff0e36df522/istio-proxy/0.log"
Oct 30 20:52:28.943691 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:28.943669    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-7vbs7_924bd04d-c63b-441d-a17b-fff0e36df522/istio-proxy/1.log"
Oct 30 20:52:28.988949 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:28.988915    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-88mz8_0cf4fb75-5c06-41d1-904b-a1b6f0784eb4/istio-proxy/0.log"
Oct 30 20:52:29.055456 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:29.055417    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-8zcb4_24bb6760-98c6-4839-bfe7-6a6a5d42dca6/istio-proxy/0.log"
Oct 30 20:52:29.543968 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:29.543934    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-86dbb48f59-9bb22_cdd067e2-0384-46e0-b85f-4580b7a107fe/istio-proxy/0.log"
Oct 30 20:52:30.499517 ip-10-0-133-26 kubenswrapper[4519]: I1030 20:52:30.499447    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-p78cx_6e04538a-4039-4b46-aba2-5fe298998516/istio-proxy/0.log"